:W&B᾿‡LM nucmi0u1ª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0ufµ€‚" zqwgna0g8ep5 nucmi0u1ŠÜ nucmi0u1escheuller-uc-san-diego spatop-sweep" _wandb‚{}  batch_size‚2048  dropout‚0.2595133647642579  hidden_dim‚128 ' learning_rate‚0.0006144894218768826  num_embedding_layers‚6Bdpfx_blocks_nucmi0u1box8or7c7j$dp-spatop-sweepagent-blocks-v2-fb276Š ÉÓõÒèÁ‰¶š3 (H¨¸hp"3.12.10*0.28.1j linux-x86_64z…Øv; nucmi0u1*  _wandb‚ {"runtime":0}  _runtime‚0¼â/data/spatop/options_files/sweep_fixed_dp/config_nucmi0u1.json*-l* /scratch/logs*-n*dpfx_blocks_nucmi0u12-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb$dp-spatop-sweepagent-blocks-v2-fb276r/opt/conda/bin/pythonº 0ott4q9i9lticvvpa8aivif6fmab9kxoæPò2  requirements.txtS¾ïW nucmi0u1ZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_64Ó¨)›Y nucmi0u1ZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64˜¯D* nucmi0u1b trainer/global_step2ö´Ÿ\  nucmi0u1ZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64ˆµ%*/  nucmi0u1b*"trainer/global_step2:ŸÚ!K  nucmi0u1j: ÊÓõÒÀªºÛ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ¥½ÉžÇ ÒÁ€@ˆ@’NVIDIA A100 80GB PCIe˜¢ /€€€€ý/€ ù¶È$ª€Àû”ÌÂS NVIDIA A100 80GB PCIe€€€€À€6"Ampere*(GPU-ed01d6ad-b673-30ac-3501-763022903663Ê13.2º 0ott4q9i9lticvvpa8aivif6fmab9kxo÷»À“  nucmi0u1j  ÊÓõÒ­…üî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 866 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 1.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 19.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ­]\ nucmi0u1jÊ ÊÓõÒðÇÅü·Trainable params: 21.2 M Non-trainable params: 50 Total params: 21.2 M Total estimated model params size (MB): 84.816 Modules in train mode: 2326 Modules in eval mode: 0 Total FLOPs: 0 í¨’q nucmi0u1j` ÊÓõÒÀ¡×üPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). ð›Ó{ nucmi0u1jj ÊÓõÒ ùÙüZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ŽÈ§¶q nucmi0u1j` ÊÓõÒˆûÛüPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update).  t nucmi0u1jc ÊÓõÒèÕÝüSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). »?19n nucmi0u1j] ÊÓõÒø ßüMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). Í6ñf. nucmi0u1*œ  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"‰¼M©ë nucmi0u1jÙ ÊÓõÒ¸¢ÿÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. n–þN nucmi0u1j¼ ÊÓõÒÐϯÿ«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. еHë nucmi0u1jÙ ÌÓõÒÈîÑØÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. (`]P nucmi0u1j¾ ÌÓõÒà•ìØ­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. »¥Â‚(:’ ÙÓõÒÔ¬„Ì disk.nvme0c0n1.in‚0$ disk.nvme0c0n1.out‚ 0.88232421875 disk.dm-1.out‚ 0.19140625( proc.memory.availableMB‚ 179.53515625 cpu‚26.435183726618657 proc.cpu.threads‚17) disk./.usagePercent‚76.21608324769925% disk./.usageGB‚1170.1074180603027 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 0.66748046875" disk.nvme1n1.out‚ 0.66748046875" disk.nvme0n1.out‚ 0.88232421875 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 0.33984375 disk.dm-0.in‚0 disk.dm-0.out‚0.625 disk.dm-1.in‚0$ memory_percent‚98.90420436859131 network.sent‚37769 network.recv‚41767# proc.memory.rssMB‚ 2825.94140625* proc.memory.percent‚17.248177528381348Ã5¹[‚(:“ ÙÓõÒ¨åÕÌ gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚47.0 gpu.0.powerWatts‚254.835' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.94500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(ÿü‚(:ˆ èÓõÒ—¡´Ë disk.dm-0.in‚0 disk.dm-1.out‚ 0.22265625 disk.rbd0.out‚0$ memory_percent‚98.90365600585938# proc.memory.availableMB‚179.625 proc.cpu.threads‚17 network.sent‚41219 disk.nvme1c1n1.in‚0# disk.nvme1c1n1.out‚ 1.2392578125 disk.md0.in‚0 disk.md0.out‚ 0.6640625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 network.recv‚45578) disk./.usagePercent‚76.21607753279697% disk./.usageGB‚1170.1073303222656! disk.nvme0n1.out‚ 9.9384765625 proc.memory.rssMB‚ 2825.96875) proc.memory.percent‚17.24834442138672 disk.dm-0.out‚ 9.5234375# disk.nvme0c0n1.out‚ 9.9384765625 disk.nvme0n1.in‚0 cpu‚26.105371337323653 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 1.2392578125‘î’‚(:ˆ èÓõÒý¬„Ì gpu.0.gpu‚64.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚49.0 gpu.0.powerWatts‚269.175' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+vjÃ3b/ lr-AdamW/pg1"trainer/global_step2:JP#©Îù3b/ lr-AdamW/pg2"trainer/global_step2:JPŒ¢²ú-) _wandbruntime‚44  _runtime‚44G’@]¿ º & lr-AdamW/pg2‚7.527495417991811e-05 trainer/global_step‚49 $ _timestamp‚1.784506870801881e+09 _runtime‚ 44.379121084 _step‚0 & lr-AdamW/pg1‚7.527495417991811e-05™9#Á!¼ & lr-AdamW/pg1‚7.527495417991811e-05 & lr-AdamW/pg2‚7.527495417991811e-05 trainer/global_step‚49 $ _timestamp‚1.784506870801881e+09 _runtime‚ 44.379121084 _step‚0rff"bb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JP/üóån#bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPkÌèft$bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPÈ…F‹f%bb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPú-ïCe&ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPðŸl'bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPÂy7_(b[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JP “L˜f)bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPÙÙï¤a*b] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPZÀa+b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPËëïs,bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPùÄoq-bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPMO†o.bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPÈ‹SIk/bg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPJIûg0bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPCµ|É_1b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JPGi2be BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JPAuº½e3ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPƒÑÍ!p4bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPf`i5be BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP¼ q›q6bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP4«RMl7bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP¼â¿m8bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPÅîqƒg9bc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JP7dÑe:ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPzû®Ri;be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JPë:Wffbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JPGŸi?be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPÏb¡&h@bd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPÈ/›sAbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPfeÜoBbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP×–›€fCbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPÒ¯o¾dDb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPä·<DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP}ýíülLbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP諲…pMbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPz-àAvNbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP1ü¹ZiObe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPÓÚhPbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPÑàŒ[iQbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPL¢û-xRbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPʼn$–gSbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPœ|vTbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPbáûcUb_ t…bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP²·i†bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP#j5šq‡bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP9˜ƒ’kˆbf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP8@¨¼p‰bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPG§ nŠbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPtjqàt‹bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPoìüœkŒbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP‘ëmoibd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPs e¨lŽbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPPwú_obj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP“pJpbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPK°‹„n‘bi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JP¸ÀŽ›h’bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JPr]•Ho“bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPf8üDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPïíÔm™bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPB‰sošbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPÏeÝh›bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP&³¨Ooœbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPÀ¥_—.) _wandbruntime‚44  _runtime‚44ï<ߺñž‚O _step‚1 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873433600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11950 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 715023360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515263 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚393553 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚119876 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 57331799040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 56563184712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 56222624966 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚212189 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7463659692888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚551 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7463302093782 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7505967331328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7407096508176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7407079468816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚491944 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7505967331328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12021 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7505609047552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7449300154880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1787262088704 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚885499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99157504 B—\LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 56650137088 QGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92313 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚478 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1844693045248 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚482152 GDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11950 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 715023360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515263 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚393553 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚119876 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 57331799040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 56563184712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 56222624966 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚212189 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7463659692888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚551 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7463302093782 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7505967331328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7407096508176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7407079468816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚491944 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7505967331328 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12021 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7505609047552 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7449300154880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1787262088704 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚885499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99157504 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPÊ:vf«ba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP’Ks'a¬b\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPê¬Úêl­bg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JP¨¾jñi®bd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPñA0l¯bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP¡ck°bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPЀŸãw±br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPOÆdÑg²bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPú:œf³ba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP0õ½Ïo´bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP¤¶#„kµbf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JP7DZ{e¶b` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPÆvZEo·bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP«µ“[b¸b] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPÌh‚g¹bb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPԇɃuºbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPGI3^»bY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JP·Ÿã®m¼bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPO$Lr½bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPOŒ»¦k¾bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPwᶑm¿bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPzFtomÀbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPg?*Ü^ÁbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP,àÝ•sÂbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPnshJ`Ãb[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPÁq4mÄbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP1%:œwÅbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP¯›vºfÆba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPqÚ;hÇbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPê4_pnÈbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JP—ÃmÉbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPöŠÓrÊbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP2ƒdípËbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP·bÌb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP`Ø©˜qÍbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP†=•mÎbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP(E]ÏbX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP¬0£9lÐbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPÛò—{kÑbf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPС OlÒbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPâ[S;_ÓbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP Š±hÔbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPÍÑÄpÕbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPK!°iÖbd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JPÁ¾²i×bd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPFÂikØbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPD=?^jÙbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPFæmÚbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPô)Å^pÛbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPHý»jÜbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPóÀ7—mÝbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPÆeAÕlÞbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPÒº—Àißbd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JPë%=B\àbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JP¹b mábh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP¤ØÖeâb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JP¥0UNkãbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JP¿»žkäbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPcKeåb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPý„+3iæbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPF¶Ìnçbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPH/.ggèbb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JPÊÜäJhébc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPåµ hêbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPËSq‘gëbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP/‰³nìbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPjï‰Îjíbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPí  fîba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPut“tVïbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP-ø¨neðb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP+± ßqñbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP_„Ÿõ_òbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPÚFcgóbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JP¤§‘›dôb_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP»µ;u€bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP¦B5Csbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP{’Û`‚b[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPSš—•eƒb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JP/× cj„be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPiI*¥h…bc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPµÍ~q†bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPk_¥j‡be BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JPâ%ülˆbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP‚ÇkØb‰b] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPÍy?jfŠba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPÜø|Tr‹bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPQÍ ûdŒb_ DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPò?i•bd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JP§oyÃd–b_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7597574657024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 57760243200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 trainer/global_step‚49 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚902962 SIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94050 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚122154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 715023360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7655351939584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚401346 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚491808 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7612198748724 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94047 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1878985536000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1878880356352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357599218 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚551 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 57418991616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7654993648640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105179648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚902962 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚122429 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 58456609792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚401346 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚216479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚478 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7597591696384 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 44.988151658 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚401348 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚216201 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7597574657024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1820442358784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7655351939584 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚491808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7597591696384 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚893154-; z–&M A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚893154 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚216201 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7597574657024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1820442358784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7655351939584 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚491808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7597591696384 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚893154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86567424 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 57671573076 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 57418991616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚401348 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 1820423746560 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 % _timestamp‚1.7845068714123752e+09 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚501614 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 58543177216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7597574657024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 57760243200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 trainer/global_step‚49 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚902962 SIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94050 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚122154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 715023360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7655351939584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚401346 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚491808 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7612198748724 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94047 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1878985536000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1878880356352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357599218 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚551 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 57418991616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7654993648640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105179648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚902962 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚122429 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 58456609792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚401346 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚216479 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚478 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7597591696384 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 44.988151658 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚401348 F loss/SRqqt1/assignment_loss"trainer/global_step2:JPV9¾A¢b< loss/FBt1/assignment_loss"trainer/global_step2:JP|6Ž‰@£b; loss/FBt1/detection_loss"trainer/global_step2:JP]S~«@¤b; loss/FBt2/detection_loss"trainer/global_step2:JPóPîýC¥b> loss/SRqqt2/assignment_loss"trainer/global_step2:JPHÆ\¸A¦b< loss/FBt2/assignment_loss"trainer/global_step2:JPÂí_O@§b; loss/FRt2/detection_loss"trainer/global_step2:JP`Ý–ÍB¨b= loss/SRqqt2/detection_loss"trainer/global_step2:JPýÍ!-©b( epoch"trainer/global_step2:JPxÁ$@ªb; loss/FRt1/detection_loss"trainer/global_step2:JPžúî7«b2 loss/total_loss"trainer/global_step2:JPø ä.¬) _wandbruntime‚44  _runtime‚44´r”ù­ó 0loss/FRt1/detection_loss‚0.40745167636511237 $loss/total_loss‚89.6145113624386 /loss/FRt1/assignment_loss‚4.529648075964218 0loss/SRqqt1/detection_loss‚5.270057057460628 /loss/FRt2/assignment_loss‚5.308231574717596 1loss/SRqqt1/assignment_loss‚4.234463186149138 0loss/FBt1/assignment_loss‚1.5068062075546809 /loss/FBt1/detection_loss‚50.960349201623885 .loss/FBt2/detection_loss‚8.819265649859346 % _timestamp‚1.7845068714157825e+09 _runtime‚ 44.991194317 1loss/SRqqt2/assignment_loss‚4.212146973006333 0loss/FBt2/assignment_loss‚1.4969060566785524 /loss/FRt2/detection_loss‚1.5700066343703853 1loss/SRqqt2/detection_loss‚1.2991790686887277 epoch‚0 trainer/global_step‚49 _step‚3dAèâý®÷ /loss/FRt2/assignment_loss‚5.308231574717596 1loss/SRqqt1/assignment_loss‚4.234463186149138 0loss/FBt1/assignment_loss‚1.5068062075546809 /loss/FBt1/detection_loss‚50.960349201623885 .loss/FBt2/detection_loss‚8.819265649859346 % _timestamp‚1.7845068714157825e+09 _runtime‚ 44.991194317 1loss/SRqqt2/assignment_loss‚4.212146973006333 0loss/FBt2/assignment_loss‚1.4969060566785524 /loss/FRt2/detection_loss‚1.5700066343703853 1loss/SRqqt2/detection_loss‚1.2991790686887277 epoch‚0 trainer/global_step‚49 _step‚3 0loss/FRt1/detection_loss‚0.40745167636511237 $loss/total_loss‚89.6145113624386 /loss/FRt1/assignment_loss‚4.529648075964218 0loss/SRqqt1/detection_loss‚5.270057057460628ÙÄÙ&¯‚(:› ÷ÓõÒÌ®¹Ë network.sent‚60327% disk./.usageGB‚1169.8697509765625 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 177.63671875 cpu‚25.928413207932163 proc.cpu.threads‚17 network.recv‚49394 disk.md0.in‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚13.72021484375# disk.nvme0n1.out‚13.72021484375! disk.nvme1n1.out‚ 1.9443359375$ memory_percent‚98.91579151153564) disk./.usagePercent‚76.20060257134425 disk.dm-0.out‚ 13.033203125 disk.dm-1.out‚ 0.49951171875 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2826.01953125 disk.md0.out‚ 0.98828125 disk.dm-0.in‚0 disk.rbd0.out‚0# disk.nvme1c1n1.out‚ 1.9443359375) proc.memory.percent‚17.24865436553955þöž°‚(:“ ÷ÓõÒþ¾Ë gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚51.0 gpu.0.powerWatts‚257.036' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.67866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò°RP±‚(:Ž †ÔõÒÖ®¤Ë disk.dm-0.in‚0 disk.rbd0.in‚0 disk.nvme0n1.out‚32.21875# proc.memory.rssMB‚ 2826.02734375 disk.md0.out‚ 1.3828125 disk.dm-1.out‚ 2.30029296875 disk.nvme0n1.in‚0$ memory_percent‚98.91407489776611( proc.memory.availableMB‚ 177.91796875 cpu‚25.831094734739338 network.recv‚55540( disk./.usagePercent‚76.2005993411821$ disk./.usageGB‚1169.869701385498! disk.nvme1n1.out‚ 6.1552734375 disk.dm-0.out‚33.07470703125 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.out‚32.21875# disk.nvme1c1n1.out‚ 6.1552734375 disk.md0.in‚0 disk.nvme0c0n1.in‚0) proc.memory.percent‚17.24870204925537 proc.cpu.threads‚17 network.sent‚99130 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0õ"=Mž²‚(:“ †ÔõÒèàûË gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚52.0 gpu.0.powerWatts‚268.286' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.42866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È÷Ïô%³‚(:š •ÔõÒ¸ê°Ë$ memory_percent‚98.91421794891357( proc.memory.availableMB‚ 177.89453125) proc.memory.percent‚17.24879741668701 network.sent‚105489% disk./.usageGB‚1169.8703575134277 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 2.51953125 disk.rbd0.out‚0 disk.nvme0n1.in‚0# disk.nvme0n1.out‚51.41943359375 network.recv‚59753) disk./.usagePercent‚76.20064207871211" disk.nvme1n1.out‚ 7.15966796875 disk.dm-0.out‚ 51.677734375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2826.04296875 cpu‚25.760145120512657 proc.cpu.threads‚17 disk.md0.out‚ 2.0390625 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 7.15966796875 disk.md0.in‚0 disk.dm-0.in‚0% disk.nvme0c0n1.out‚51.41943359375Äød’´‚(:‡ •ÔõÒçùúË gpu.0.gpu‚24.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚53.0 gpu.0.powerWatts‚283.305' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™¯þ!.µ) _wandbruntime‚86  _runtime‚86çÙyÚö½ ' lr-AdamW/pg1‚0.00015208613191452843 ' lr-AdamW/pg2‚0.00015208613191452843 trainer/global_step‚99 % _timestamp‚1.7845069129610083e+09 _runtime‚ 86.537240921 _step‚4U›þÇ·Á trainer/global_step‚99 % _timestamp‚1.7845069129610083e+09 _runtime‚ 86.537240921 _step‚4 ' lr-AdamW/pg1‚0.00015208613191452843 ' lr-AdamW/pg2‚0.00015208613191452843}E}.¸) _wandbruntime‚86  _runtime‚86–ú@ƒ£'¹O J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚783203 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1750007 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 112219238912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚179013 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3559480057344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873434624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚5 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3446256980480 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 112219238912 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚414742 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚99 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1759801 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚783205 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚179016 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704150119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 111877994496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 112046398120 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14863042301440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚577 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14778612842416 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚976596 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14974903256576 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14890318680790 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚783205 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14863042301440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3446275592704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚235729 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180q†~- º¡O MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14975261540352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 113334817280 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14890676279896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 113223076864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚ 86.537304941 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚783203 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14778629881776 % _timestamp‚1.7845069129616988e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14863025262080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111740416 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚966804 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚415043 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14890318680790 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚783205 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14863042301440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3446275592704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚235729 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚783203 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1750007 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 112219238912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚179013 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3559480057344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873434624 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚5 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3446256980480 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 112219238912 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚414742 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚99 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1759801 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚783205 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚179016 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704150119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 111877994496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 112046398120 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14863042301440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚577 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14778612842416 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚976596 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14974903256576 HDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1777264 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚976460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115927552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14926043509888 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1767456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚ 87.152521659 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚790998 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134539776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3479455862784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 113329345024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15011316803584 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚180750 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14926060549248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3479437250560 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 113154786484 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15011299764224 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 113329345024 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚419039 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3593772686336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 114451363328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 $ _timestamp‚1.784506913576712e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 112988093440 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1767456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15038857736514 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚418733 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚577 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15011299764224 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 113329345024 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚419039 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3593772686336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 114451363328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 $ _timestamp‚1.784506913576712e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 112988093440 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1767456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15038857736514 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚418733 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚577 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1777264 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚976460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115927552 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14926043509888 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1767456 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚ 87.152521659 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚790998 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134539776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3479455862784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 113329345024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15011316803584 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚180750 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14926060549248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3479437250560 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 113154786484 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22150146216016 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 167788340736 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚351468 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚617181 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 167189051782 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _step‚9 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚617473 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚504 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 5274345911296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2624309 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22444197465600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22150163255376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1172855 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22444197465600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5105270484480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 169178778624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845069550850804e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22276767408640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5274467875328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 167529611528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚ 128.662111802 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5105289096704 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22276750369280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1172853 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22276767408640 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 167788340736 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22276750369280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1172853 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22276767408640 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 167788340736 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22150146216016 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 167788340736 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚351468 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚617181 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 167189051782 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _step‚9 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚617473 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚504 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 5274345911296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2624309 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22444197465600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22150163255376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1172855 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22444197465600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5105270484480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 169178778624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1451456 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845069550850804e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22276767408640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5274467875328 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 j;0Í+ _wandbruntime‚129  _runtime‚129wã&βM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚621197 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1461112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22425041910784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845069556966138e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22425041910784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22593940357632 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1180648 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5308639998464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22297576883488 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2641758 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1470918 _step‚10 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22593582066688 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1180646 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚353750 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5138450754560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1180648 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 168557195264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2651566 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357599218 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22466231922740 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5308766149632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚267447 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚354040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22425024871424 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 170296782848 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 170189243904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22593940357632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 117210624Cx¤$¼Ï¶M ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5308766149632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚267447 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚354040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22425024871424 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 170296782848 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 170189243904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22593940357632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 117210624 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚621197 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1461112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22425041910784 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845069556966138e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22425041910784 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22593940357632 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1180648 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5308639998464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22297576883488 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2641758 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1470918 _step‚10 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22593582066688 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1180646 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚353750 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5138450754560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1180648 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 168557195264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2651566 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357599218 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22466231922740 e]F0Ð+ _wandbruntime‚129  _runtime‚129ý ·”ÿÑù /loss/FRt1/assignment_loss‚4.510361919112483 1loss/FBt2/assignment_loss‚0.19265921951531104 1loss/SRqqt2/detection_loss‚1.0129711559942625 .loss/FBt2/detection_loss‚6.421032044978172 _runtime‚ 129.273676078 /loss/FRt2/assignment_loss‚4.676175687354424 1loss/SRqqt1/assignment_loss‚2.485723206366616 trainer/global_step‚149 2loss/SRqqt2/assignment_loss‚2.2018449378787044 0loss/FRt1/detection_loss‚0.30609569037999856 .loss/FBt1/detection_loss‚20.18361974939228 epoch‚0 % _timestamp‚1.7845069556983612e+09 1loss/FBt1/assignment_loss‚0.24671735435291456 /loss/FRt2/detection_loss‚1.5818414951895359 0loss/SRqqt1/detection_loss‚4.697029437684236 %loss/total_loss‚48.51607189819894 _step‚11œl7œÒý _step‚11 /loss/FRt1/assignment_loss‚4.510361919112483 1loss/FBt2/assignment_loss‚0.19265921951531104 1loss/SRqqt2/detection_loss‚1.0129711559942625 .loss/FBt2/detection_loss‚6.421032044978172 _runtime‚ 129.273676078 /loss/FRt2/assignment_loss‚4.676175687354424 1loss/SRqqt1/assignment_loss‚2.485723206366616 trainer/global_step‚149 2loss/SRqqt2/assignment_loss‚2.2018449378787044 0loss/FRt1/detection_loss‚0.30609569037999856 .loss/FBt1/detection_loss‚20.18361974939228 epoch‚0 % _timestamp‚1.7845069556983612e+09 1loss/FBt1/assignment_loss‚0.24671735435291456 /loss/FRt2/detection_loss‚1.5818414951895359 0loss/SRqqt1/detection_loss‚4.697029437684236 %loss/total_loss‚48.51607189819894 ­\µžÓ‚(:“ ÑÔõÒö›ÎË gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚273.232' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.07733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aê“(Ô‚(: ÑÔõÒ§ì³Ì network.sent‚167259 disk.dm-1.out‚ 3.2294921875 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.md0.out‚ 3.37890625 cpu‚25.620189969593724 proc.cpu.threads‚17 network.recv‚76071 disk.dm-1.in‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 275.986328125 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0# memory_percent‚98.9072322845459) disk./.usagePercent‚76.20057474225493% disk.nvme1c1n1.out‚43.00146484375 disk.dm-0.in‚0! disk.dm-0.out‚308.47998046875' proc.memory.availableMB‚ 179.0390625* proc.memory.percent‚17.248821258544922% disk./.usageGB‚1169.8693237304688 disk.rbd0.out‚0" disk.nvme0n1.out‚ 275.986328125# disk.nvme1n1.out‚43.00146484375! proc.memory.rssMB‚ 2826.046875-„CY'Õ‚(:œ àÔõÒ†ãÆË) disk./.usagePercent‚76.20889191285075 disk.nvme0n1.in‚0% disk.nvme1c1n1.out‚50.04052734375 cpu‚25.603601638193854 network.recv‚80272 disk.nvme1c1n1.in‚0# disk.nvme1n1.out‚50.04052734375 disk.md0.out‚ 5.59765625 disk.dm-0.out‚358.7880859375 disk.dm-1.in‚0 disk.dm-1.out‚ 3.349609375 disk.rbd0.in‚0 network.sent‚216092$ disk./.usageGB‚1169.997013092041 disk.dm-0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚323.88525390625# memory_percent‚98.9084243774414* proc.memory.percent‚17.248821258544922$ disk.nvme0n1.out‚323.88525390625 disk.nvme1n1.in‚0 disk.md0.in‚0% proc.memory.availableMB‚ 178.84375! proc.memory.rssMB‚ 2826.046875 proc.cpu.threads‚17NvßžÖ‚(:“ àÔõÒ“žŒÌ gpu.0.gpu‚36.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚280.596' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.53200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:Wû¡×‚(:’ ïÔõҶ¶Ë! disk.dm-0.out‚373.94384765625 disk.dm-1.out‚3.75$ memory_percent‚98.90909194946289& proc.memory.availableMB‚ 178.734375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0$ disk.nvme0c0n1.out‚ 339.509765625 disk.nvme1c1n1.in‚0# disk.nvme1n1.out‚51.27685546875! proc.memory.rssMB‚ 2826.046875* proc.memory.percent‚17.248821258544922 proc.cpu.threads‚17 network.sent‚221471% disk./.usageGB‚1170.1417770385742 disk.nvme1n1.in‚0 disk.md0.out‚6.21875 disk.dm-0.in‚0 disk.dm-1.in‚0 cpu‚25.590925420846943) disk./.usagePercent‚76.21832125312322 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 339.509765625 network.recv‚84086 disk.rbd0.in‚0% disk.nvme1c1n1.out‚51.27685546875 disk.md0.in‚0Ãÿ›Ø‚(: ïÔõÒõÂþË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚166.81' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚55.60333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Nl#0Ù+ _wandbruntime‚170  _runtime‚170åç54ÆÚÀ ' lr-AdamW/pg1‚0.00030570848738374905 ' lr-AdamW/pg2‚0.00030570848738374905 trainer/global_step‚199 % _timestamp‚1.7845069970753267e+09 _runtime‚ 170.650942416 _step‚12°/ÊÛÄ ' lr-AdamW/pg2‚0.00030570848738374905 trainer/global_step‚199 % _timestamp‚1.7845069970753267e+09 _runtime‚ 170.650942416 _step‚12 ' lr-AdamW/pg1‚0.00030570848738374905 9n€40Ü+ _wandbruntime‚170  _runtime‚170ì!Í­'ݧO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29690492515840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6989325793792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29744351854806 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29690475476480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29690475476480 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚467156 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 223012824936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1945900 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚352413 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29744709453912 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29521679589616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1936108 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873434624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚13 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29913491674624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3508405 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29521696628976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105448960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 222672265190 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚352416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚466879 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 124061184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6764302600704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 $ _timestamp‚1.784506997076339e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 224917744128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚819572 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _runtime‚ 170.655067514 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29913849958400 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 223016198144IÇ~#Þ«O XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚819572 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _runtime‚ 170.655067514 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚29913849958400 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 223016198144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29690492515840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6989325793792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29744351854806 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29690475476480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29690475476480 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚467156 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 223012824936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1945900 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚352413 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29744709453912 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29521679589616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1936108 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873434624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚13 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29913491674624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3508405 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29521696628976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105448960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 222672265190 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚352416 UGDeviceStatsMonitor.on_train_batch_start/activ9°_3e_bytes.small_pool.current‚ 341244416 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚466879 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 124061184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6764302600704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 $ _timestamp‚1.784506997076339e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 224917744128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XÃî/0ß+ _wandbruntime‚171  _runtime‚171ÃÍC.¸&à²M MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1945764 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 224467548672 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚823571 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _runtime‚ 171.260459359 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3525868 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚354147 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1570298 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 223780653442 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29838749978624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 224126297088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚354150 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3525868 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30063234566656 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚469122 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7845069976846318e+09 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 224467548672 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚469421 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7023490273792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29838767017984 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29893248509748 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29669110257088 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3516060 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚199 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 224121213300 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30062876275712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3516060 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130345472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29838749978624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29838767017984 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚823269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6797482870784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6797464258560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111733248 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _step‚14 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1570296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1570296 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1945764 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29669127296448 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 224126297088 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 224467548672 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚823571 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _runtime‚ 171.260459359 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3525868 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚354147 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1570298 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 223780653442 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29838749978624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 224126297088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚354150 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3525868 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30063234566656 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚469122 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7845069976846318e+09 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 224467548672 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚469421 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7023490273792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29838767017984 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚29893248509748 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29669110257088 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3516060 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚199 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 224121213300 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30062876275712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3516060 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130345472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29838749978624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29838767017984 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚823269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6797482870784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6797464258560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111733248 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _step‚14 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1570296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1570296 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1945764 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29669127296448 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 224126297088 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37382785883648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36893230002576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚439113 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 280742208000 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2430552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36893212963216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37104217623040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1021452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2430552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 278926544384 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 278585299968 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37104217623040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 280849754112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2420760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 278585299968 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37383144167424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚17 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1952155 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚582622 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚582339 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107546112 _runtime‚ 212.536110776 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704150119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8423297492480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1952153 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278926544384 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1952153 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126158336/úò³'ð­O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 280849754112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2420760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 278585299968 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37383144167424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚17 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1952155 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚582622 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚582339 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107546112 _runtime‚ 212.536110776 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704150119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8423297492480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1952153 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278926544384 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1952153 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126158336 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1952155 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37382785883648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36893230002576 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚439113 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 280742208000 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2430552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36893212963216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚37104217623040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1021452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2430552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 278926544384 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 278585299968 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚37104217623040óšs 0ñ+ _wandbruntime‚213  _runtime‚213†v¨óº&ò´M P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1959946 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873434624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1959948 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845070395707605e+09 _runtime‚ 213.146557583 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4390362 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 279604426708 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4390362 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37532528775680 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚440850 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8456477762560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37252475085824 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1959948 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8738336339456 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37320265096756 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37319907497538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚584588 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8456496374784 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 281961921536 _step‚18 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 280036650496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 279695398912 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2440222 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37252492125184 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 280036650496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1959946 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 279695398912 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4400170 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚440847 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8738458296320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2430416 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 281858576896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357599218 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚584877 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37532528775680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2440222 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37040660670048 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 279263866850 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4400170 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1025435».²ÝÑ ó¸M TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚440847 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8738458296320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2430416 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 281858576896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357599218 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚584877 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37532528775680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2440222 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37040660670048 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 279263866850 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4400170 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1025435 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1959946 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873434624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1959948 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845070395707605e+09 _runtime‚ 213.146557583 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4390362 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 279604426708 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4390362 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37532528775680 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚440850 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8456477762560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37252475085824 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1959948 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8738336339456 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37320265096756 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37319907497538 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚584588 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8456496374784 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 281961921536 _step‚18 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 280036650496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 279695398912 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2440222 OA°ÝÈíDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37252492125184 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 280036650496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1959946 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12023 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 279695398912 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4400170 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184CºŸ0ô+ _wandbruntime‚213  _runtime‚213-NÛ–ÿõù 1loss/FBt2/assignment_loss‚0.19845147422606113 1loss/SRqqt1/detection_loss‚2.6967714684574227 1loss/SRqqt2/detection_loss‚1.3038802738894213 _runtime‚ 213.148880097 epoch‚0 /loss/FRt2/assignment_loss‚4.572891386174366 /loss/FRt2/detection_loss‚1.4378114899064707 $loss/total_loss‚36.3012979279944 trainer/global_step‚249 % _timestamp‚1.7845070395735717e+09 /loss/FRt1/assignment_loss‚4.517890329937727 2loss/SRqqt2/assignment_loss‚2.2284286129729827 /loss/FRt1/detection_loss‚0.4138867106606235 .loss/FBt1/detection_loss‚9.418229110669817 .loss/FBt2/detection_loss‚7.796322104759775 _step‚19 2loss/SRqqt1/assignment_loss‚1.5704167417985828 1loss/FBt1/assignment_loss‚0.14631822454114704l¤uöý /loss/FRt1/assignment_loss‚4.517890329937727 2loss/SRqqt2/assignment_loss‚2.2284286129729827 /loss/FRt1/detection_loss‚0.4138867106606235 .loss/FBt1/detection_loss‚9.418229110669817 .loss/FBt2/detection_loss‚7.796322104759775 _step‚19 2loss/SRqqt1/assignment_loss‚1.5704167417985828 1loss/FBt1/assignment_loss‚0.14631822454114704 1loss/FBt2/assignment_loss‚0.19845147422606113 1loss/SRqqt1/detection_loss‚2.6967714684574227 1loss/SRqqt2/detection_loss‚1.3038802738894213 _runtime‚ 213.148880097 epoch‚0 /loss/FRt2/assignment_loss‚4.572891386174366 /loss/FRt2/detection_loss‚1.4378114899064707 $loss/total_loss‚36.3012979279944 trainer/global_step‚249 % _timestamp‚1.7845070395735717e+09«žìA)÷‚(:ž «ÕõÒî×åË disk.nvme0n1.in‚0$ disk.nvme0n1.out‚573.82177734375 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚121.0712890625) disk./.usagePercent‚76.21842287899403% disk./.usageGB‚1170.1433372497559 disk.md0.in‚0 disk.dm-1.out‚ 4.94482421875 disk.nvme1n1.in‚0$ memory_percent‚98.91605377197266! proc.memory.rssMB‚ 2826.046875* proc.memory.percent‚17.248821258544922 network.sent‚294903 disk.md0.out‚ 11.65234375 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚573.82177734375# disk.nvme1n1.out‚121.0712890625 cpu‚25.555789431538653 network.recv‚99645 disk.rbd0.out‚0% proc.memory.availableMB‚ 177.59375 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.dm-0.out‚ 665.7265625 disk.dm-1.in‚0 disk.rbd0.in‚0IQЮ‘ø‚(:† «ÕõÒú‹£Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.92' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.64 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Îåí¼-ù‚(:¢ ºÕõÒÖ™ÜË disk.nvme0n1.in‚0' proc.memory.availableMB‚ 177.5703125* proc.memory.percent‚17.248821258544922 network.sent‚330814 network.recv‚103905 disk.dm-0.in‚0! disk.dm-0.out‚703.11474609375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme0c0n1.out‚610.9326171875% disk.nvme1c1n1.out‚122.4619140625 disk.md0.in‚0 disk.md0.out‚ 12.16015625! proc.memory.rssMB‚ 2826.046875 cpu‚25.54668450907218 proc.cpu.threads‚17) disk./.usagePercent‚76.21838511094424# disk.nvme0n1.out‚610.9326171875# disk.nvme1n1.out‚122.4619140625$ memory_percent‚98.91619682312012% disk./.usageGB‚1170.1427574157715 disk.dm-1.in‚0 disk.dm-1.out‚ 4.97998046875 disk.rbd0.in‚0غ+âžú‚(:“ ºÕõÒîŸÌ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚269.234' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.74466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ëpÜ0û+ _wandbruntime‚254  _runtime‚254Èè­†Åü¿ _runtime‚ 254.457058799 _step‚20 ' lr-AdamW/pg1‚0.00045933084285296975 ' lr-AdamW/pg2‚0.00045933084285296975 trainer/global_step‚299 $ _timestamp‚1.784507080879605e+09ÝI)Éýà _step‚20 ' lr-AdamW/pg1‚0.00045933084285296975 ' lr-AdamW/pg2‚0.00045933084285296975 trainer/global_step‚299 $ _timestamp‚1.784507080879605e+09 _runtime‚ 254.457058799–ËœË0þ+ _wandbruntime‚254  _runtime‚254_³´'ÿ®O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚299 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 334154401792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44598385028822 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10418940256768 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5257009 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚525816 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚697878 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10082329608704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 333979251752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44517942730240 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 336738903552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44852080092672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44517925690880 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2341803 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128255488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚44264746336816 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚21 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 334154401792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44264763376176 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 333638692006 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚525813 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5247215 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚698167 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873434624 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2341805 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44852438376448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 334495646208 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 334495646208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44517942730240 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10082310996480 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44598742627928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845070808802667e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2341803 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1223983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44852438376448 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44852080092672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 336629260288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2915204 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2915204 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2905412 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3göj»¸'€²O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 334154401792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚44264763376176 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 333638692006 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚525813 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5247215 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚698167 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873434624 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2341805 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358283776 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44852438376448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 334495646208 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 334495646208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44517942730240 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10082310996480 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44598742627928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845070808802667e+09 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2341803 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1223983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44852438376448 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44852080092672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 336629260288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2915204 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358283776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2915204 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2905412 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚299 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 334154401792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44598385028822 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10418940256768 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5257009 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚525816 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚697878 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10082329608704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 333979251752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44517942730240 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 336738903552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44852080092672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44517925690880 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2341803 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128255488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚44264746336816 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚21²ï£ö%‚(:š ÉÕõÒÇÜÓË network.sent‚347236 disk.rbd0.out‚0! proc.memory.rssMB‚ 2826.046875* proc.memory.percent‚17.248821258544922) disk./.usagePercent‚76.23386504187721# disk./.usageGB‚1170.38041305542 disk.nvme1c1n1.in‚0! disk.dm-0.out‚953.18896484375 disk.dm-1.out‚ 5.23388671875# disk.nvme0c0n1.out‚ 829.18359375( proc.memory.availableMB‚ 177.33203125 proc.cpu.threads‚17 disk.dm-0.in‚0 disk.nvme0n1.in‚0! disk.nvme0n1.out‚ 829.18359375$ disk.nvme1c1n1.out‚ 155.689453125 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 155.689453125 disk.md0.in‚0 disk.md0.out‚ 12.703125 network.recv‚108697 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0$ memory_percent‚98.91765117645264 cpu‚25.545037785799103H7‚‚(:’ ÉÕõÒæÌ gpu.0.gpu‚92.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚266.08' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.69333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÈŽ0ƒ+ _wandbruntime‚255  _runtime‚2555±¡¬’„¸M H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 335264500736 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚527550 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _step‚22 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10453234327552 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2349598 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2349596 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45001822984704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357599218 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45001822984704 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2349598 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2915068 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113830400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚299 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5274472 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5264664 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚700413 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44666217232384 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45001464693760 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 335264500736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873434624 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 335605752320 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2349596 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10115509878784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2915068 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44666200193024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1227662 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1011549ß“,1266560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2924874 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44666217232384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44747281683764 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 335605752320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44412194043648 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5274472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 335087640116 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10453366770176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44412177004288 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 334747080258 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2915068 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113830400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚299 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5274472 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5264664 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚700413 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44666217232384 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45001464693760 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 335264500736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873434624 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 335605752320 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2349596 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10115509878784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2915068 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44666200193024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1227662 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12023 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10115491266560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2924874 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44666217232384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44747281683764 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 335605752320 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44412194043648 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5274472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 335087640116 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10453366770176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚44412177004288 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 334747080258 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51931650798080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11741324500480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704150119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 % _timestamp‚1.7845071229187598e+09 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 390064748032 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3399856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111740416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52321732585472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51931667837440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚25 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚813298 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51636296749776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1426098 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130352640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 390064748032 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3390064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52321374301696 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1425811 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12133910036480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52025759214936 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6121517 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2731453 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6131311 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2731455 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚612516 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 392566923776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚349 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚119523ÀàëG ’²O D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6121517 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2731453 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6131311 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2731455 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚612516 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 392566923776 GADeviceStatsMonitorïÂ3q.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚349 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2731455 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3390064 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52321732585472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51931650798080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11741324500480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704150119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 % _timestamp‚1.7845071229187598e+09 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 390064748032 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3399856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111740416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52321732585472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51931667837440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚25 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚813298 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51636296749776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560515271 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1426098 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130352640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 390064748032 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3390064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52321374301696 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1425811 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚12133910036480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52025759214936•t’Ý0“+ _wandbruntime‚297  _runtime‚297Q ý±¾&”¸M F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6138966 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚349 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52471117193728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51783727417248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107538944 _step‚26 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12168204295680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚52079925300224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 390833602560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6148774 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126151168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1430127 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12168078144512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3409526 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2739248 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚614247 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3399720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 390570853524 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 390833602560 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845071235308561e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 391174854144 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 391174854144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2739248 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52470758902784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1429838 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 390230293666 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2739246 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚52079942339584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784UÁ$ Â&•¼M OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358290944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12168204295680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚52079925300224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 390833602560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6148774 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 126151168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1430127 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12168078144512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3409526 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2739248 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚614247 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9808 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3399720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 390570853524 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 390833602560 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845071235308561e+09 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 391174854144 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 391174854144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9806 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2739248 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52470758902784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1429838 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9806 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 390230293666 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2739246 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚52079942339584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 119307776 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6138966 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚349 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52471117193728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51783727417248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561206784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107538944 _step‚26%T~Ê0–+ _wandbruntime‚297  _runtime‚297µý',ý—÷ epoch‚0 _runtime‚ 297.107093292 /loss/FRt2/assignment_loss‚3.621141080163483 0loss/SRqqt1/assignment_loss‚1.76200566629443 0loss/FBt2/assignment_loss‚0.2399015549140953 &loss/total_loss‚28.670232310428165 % _timestamp‚1.7845071235318425e+09 /loss/FRt1/detection_loss‚0.4682196679727357 .loss/FBt2/detection_loss‚6.534236054822623 trainer/global_step‚349 1loss/FBt1/assignment_loss‚0.22901585253795037 /loss/FRt2/detection_loss‚1.3880495920623428 0loss/SRqqt1/detection_loss‚2.984910216654026 _step‚27 /loss/FRt1/assignment_loss‚4.046979690751722 2loss/SRqqt2/assignment_loss‚1.5676337104251092 1loss/SRqqt2/detection_loss‚0.9426565238750203 .loss/FBt1/detection_loss‚4.885482699954626¡¥tü˜û /loss/FRt2/assignment_loss‚3.621141080163483 0loss/SRqqt1/assignment_loss‚1.76200566629443 0loss/FBt2/assignment_loss‚0.2399015549140953 &loss/total_loss‚28.670232310428165 % _timestamp‚1.7845071235318425e+09 /loss/FRt1/detection_loss‚0.4682196679727357 .loss/FBt2/detection_loss‚6.534236054822623 trainer/global_step‚349 1loss/FBt1/assignment_loss‚0.22901585253795037 /loss/FRt2/detection_loss‚1.3880495920623428 0loss/SRqqt1/detection_loss‚2.984910216654026 _step‚27 /loss/FRt1/assignment_loss‚4.046979690751722 2loss/SRqqt2/assignment_loss‚1.5676337104251092 1loss/SRqqt2/detection_loss‚0.9426565238750203 .loss/FBt1/detection_loss‚4.885482699954626 epoch‚0 _runtime‚ 297.107093292Ð ²/™‚(:¤ öÕõÒòú·Ë disk.md0.out‚ 14.19921875 disk.dm-0.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 177.02734375) disk./.usagePercent‚76.22174994601156% disk./.usageGB‚1170.1944160461426 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚865.26611328125$ disk.nvme1c1n1.out‚ 211.044921875 disk.dm-0.out‚989.9912109375 disk.dm-1.out‚56.67626953125 disk.rbd0.out‚0 network.recv‚120774" disk.nvme1n1.out‚ 211.044921875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0! proc.memory.rssMB‚ 2826.046875 proc.cpu.threads‚17 network.sent‚412474 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0& disk.nvme0c0n1.out‚865.26611328125$ memory_percent‚98.91951084136963* proc.memory.percent‚17.248821258544922 cpu‚25.539455897294715äOœš‚(:‘ öÕõÒÉñŸÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.926' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.97533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô^‚™/›‚(:¤ …ÖõÒ²²ÁË proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚876.35205078125 disk.nvme1n1.in‚0! disk.dm-0.out‚1000.6787109375 disk.dm-1.in‚0( proc.memory.availableMB‚ 176.93359375" disk.nvme1n1.out‚ 213.978515625 disk.md0.in‚0 disk.md0.out‚ 14.64453125* proc.memory.percent‚17.248821258544922 cpu‚25.53765718202169 network.sent‚461242 network.recv‚125470) disk./.usagePercent‚76.22280993614575% disk./.usageGB‚1170.2106895446777 disk.rbd0.in‚0$ disk.nvme1c1n1.out‚ 213.978515625 disk.dm-1.out‚59.03955078125! proc.memory.rssMB‚ 2826.046875 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚876.35205078125 disk.nvme1c1n1.in‚0 disk.dm-0.in‚0$ memory_percent‚98.92008304595947FüÖÔŸœ‚(:” …ÖõÒö̆Ì gpu.0.gpu‚87.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.317' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.105666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßÍ-(‚(: ”ÖõÒÌŸ¶Ë network.recv‚129026) disk./.usagePercent‚76.22385327852116 disk.dm-0.in‚0 disk.dm-0.out‚ 1015.8125 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚891.59130859375$ disk.nvme1c1n1.out‚ 214.888671875 disk.dm-1.out‚59.12548828125 disk.nvme0c0n1.in‚0 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 214.888671875 disk.md0.in‚0$ memory_percent‚98.92022609710693( proc.memory.availableMB‚ 176.91015625* proc.memory.percent‚17.248821258544922 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚891.59130859375 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.md0.out‚ 15.0703125 cpu‚25.536039020352554 proc.cpu.threads‚17 network.sent‚466397$ disk./.usageGB‚1170.226707458496! proc.memory.rssMB‚ 2826.046875¼'M´žž‚(:“ ”ÖõÒŽüüË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚162.693' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚54.23100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨"ï¾0Ÿ+ _wandbruntime‚338  _runtime‚338ïíí1Ä ¾ _step‚28 & lr-AdamW/pg1‚0.0006129531983221904 & lr-AdamW/pg2‚0.0006129531983221904 trainer/global_step‚399 % _timestamp‚1.7845071650254202e+09 _runtime‚ 338.6010257568˜èÈ¡Â _runtime‚ 338.601025756 _step‚28 & lr-AdamW/pg1‚0.0006129531983221904 & lr-AdamW/pg2‚0.0006129531983221904 trainer/global_step‚399 % _timestamp‚1.7845071650254202e+09÷ù¢+ _wandbruntime‚338 <{ON _runtime‚338P€n´'£®O \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59345392944640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105448960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚699213 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13848646689792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 444605118822 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 448308685312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 445292605440 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚928662 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3121103 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚399 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59791026794496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59345392944640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 338.605052235 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3121103 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 124061184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 445633849856 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59007830123376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3884508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 445292605440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3121105 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 448414134272 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13400356616704 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59345375905280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3121105 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3874716 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59452418202838 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1628173 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚699216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13848770750976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1627875 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59790668510720 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59790668510720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚928957 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 % _timestamp‚1.7845071650263836e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59007813084016 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3874716 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59791026794496 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7005613 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59345375905280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59452775801944 _step‚29ÅÛˆ¸'¤²O UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59791026794496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59345392944640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 119307776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚ 338.605052235 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3121103 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 124061184 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357599106 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 445633849856 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341244416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59007830123376 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3884508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873434624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9794 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 445292605440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340559746 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3121105 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 448414134272 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9794 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13400356616704 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59345375905280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341244416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12023 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3121105 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3874716 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59452418202838 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1628173 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚699216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13848770750976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1627875 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59790668510720 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59790668510720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11952 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚928957 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 % _timestamp‚1.7845071650263836e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59007813084016 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3874716 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59791026794496 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7005613 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9792 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59345375905280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59452775801944 _step‚29 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59345392944640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561206784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105448960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚699213 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13848646689792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 444605118822 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 448308685312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561206784 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 445292605440 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚928662 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12023 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3121103 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3999(¼R0¥+ _wandbruntime‚339  _runtime‚339{–õ¾&¦¸M N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 446402704384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1631914 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚930967 _step‚30 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7013268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59940053111808 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873434624 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7023076 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3884372 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3128896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59601314857780 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3128896 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 446054066932 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 445713507074 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚399 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3884372 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13882947449344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3894178 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59940411402752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 449532519424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59155243751488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 449429174784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59600957258562 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13433518274560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59493650407424 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13883069406208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271C m k §¼M MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3884372 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3128896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59601314857780 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3128896 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 446054066932 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 445713507074 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358290944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚399 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3884372 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13882947449344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3894178 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59940411402752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341251584 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 449532519424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59155243751488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 449429174784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340559858 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 R?DeviceStatsMonito› ¥RWr.on_train_batch_end/requested_bytes.all.freed‚59600957258562 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13433518274560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561206784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59493650407424 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13883069406208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560515271 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873434624 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 446402704384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1631914 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚930967 _step‚30 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7013268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704150119 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341251584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59940053111808 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873434624 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7023076 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2#_z¸0¨+ _wandbruntime‚339  _runtime‚339Ÿg5ÿ©ù /loss/FRt2/assignment_loss‚3.438522871891102 1loss/FBt2/assignment_loss‚0.41667592828162014 /loss/FRt2/detection_loss‚1.3928904711985244 epoch‚0 trainer/global_step‚399 % _timestamp‚1.7845071656398895e+09 1loss/SRqqt1/assignment_loss‚1.536092300490852 /loss/FRt1/detection_loss‚0.5093043727641342 &loss/total_loss‚28.081423493261372 _runtime‚ 339.215237265 /loss/FRt1/assignment_loss‚3.864357816943124 2loss/SRqqt2/assignment_loss‚1.5177606098907743 1loss/FBt1/assignment_loss‚0.12977667171955856 0loss/SRqqt1/detection_loss‚2.973504585981892 1loss/SRqqt2/detection_loss‚0.8113603802415714 .loss/FBt1/detection_loss‚2.279834207357515 _step‚31 .loss/FBt2/detection_loss‚9.211343276500703¬íS§ªý .loss/FBt2/detection_loss‚9.211343276500703 /loss/FRt2/assignment_loss‚3.438522871891102 1loss/FBt2/assignment_loss‚0.41667592828162014 /loss/FRt2/detection_loss‚1.3928904711985244 epoch‚0 trainer/global_step‚399 % _timestamp‚1.7845071656398895e+09 1loss/SRqqt1/assignment_loss‚1.536092300490852 /loss/FRt1/detection_loss‚0.5093043727641342 &loss/total_loss‚28.081423493261372 _runtime‚ 339.215237265 /loss/FRt1/assignment_loss‚3.864357816943124 2loss/SRqqt2/assignment_loss‚1.5177606098907743 1loss/FBt1/assignment_loss‚0.12977667171955856 0loss/SRqqt1/detection_loss‚2.973504585981892 1loss/SRqqt2/detection_loss‚0.8113603802415714 .loss/FBt1/detection_loss‚2.279834207357515 _step‚31[sÌÖ&«‚(:› £ÖõÒظ°Ë disk.nvme0c0n1.in‚0 disk.md0.in‚0" disk.nvme1n1.out‚ 216.568359375# proc.memory.rssMB‚ 2826.16015625 cpu‚25.573998992393122) disk./.usagePercent‚76.22589200778769 disk.dm-0.in‚0 disk.dm-0.out‚ 1037.40234375 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 216.568359375 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 913.388671875 network.sent‚469715 network.recv‚132855% disk./.usageGB‚1170.2580070495605 disk.md0.out‚ 15.5546875 disk.dm-1.in‚0 disk.dm-1.out‚ 59.98828125 disk.rbd0.in‚0" disk.nvme0n1.out‚ 913.388671875 disk.nvme1n1.in‚0# memory_percent‚98.9220142364502' proc.memory.availableMB‚ 176.6171875* proc.memory.percent‚17.249512672424316 proc.cpu.threads‚17tØΞ¬‚(:“ £ÖõÒ®×´Ì gpu.0.gpu‚88.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.077' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.35900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯$#¨J­bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPgBUŒJ®bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPP? ª@¯b; particle/accuracy_1_of_4"trainer/global_step2:JP”àJ°bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPïwÒÿH±bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPžO4"H²bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP`-šN³bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPÊ5WÊH´bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPf#9*JµbE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÏp6ÕN¶bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP/&;·b6 jet/accuracy_1_of_2"trainer/global_step2:JP1 ¡#H¸bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPS»…ºJ¹bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÚ”ªCJºbE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP"{¸ƒN»bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPä¬ïÈN¼bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÉ åH½bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPG:OïN¾bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÝ5«‡H¿bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPÖýuÈHÀbC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPuìJHÁbC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPºqÍ”@Âb; particle/accuracy_1_of_5"trainer/global_step2:JP t ·HÃbC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPŽjžNÄbI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPfÖÈHÅbC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPYŽ;Æb6 jet/accuracy_3_of_4"trainer/global_step2:JPj_8Çb3 particle/f_score"trainer/global_step2:JP)­jHÈbC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPî‹÷ JÉbE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPùª_BNÊbI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP@UÃJËbE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP³8*âHÌbC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP ²ë HÍbC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP4(¿uJÎbE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP¦S} JÏbE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¬÷èŠ;Ðb6 jet/accuracy_4_of_5"trainer/global_step2:JP÷<%þJÑbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÎ:X¿JÒbE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPFé»èHÓbC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPgyÙ‡HÔbC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP‘À;Õb6 jet/accuracy_4_of_4"trainer/global_step2:JPè$€¿@Öb; particle/accuracy_3_of_4"trainer/global_step2:JP<(ªJ×bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPrC¼HØbC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP¨³¾(NÙbI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP«ÂÈÃNÚbI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPèCɇ;Ûb6 jet/accuracy_3_of_3"trainer/global_step2:JP¦Œ\ÄJÜbE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPqÈGNÝbI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPkÕJÞbE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPø%æNHßbC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPÔS£òJàbE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP}IrÖNábI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPS×!;âb6 jet/accuracy_2_of_3"trainer/global_step2:JPå2rJãbE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÁ5ÀÂJäbE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPxÏ¢–HåbC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPømNæbI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPÛÀ>HçbC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPc§¼d@èb; particle/accuracy_4_of_5"trainer/global_step2:JPFIرHébC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPàBúÊHêbC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÆ'ÿñJëbE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¤GC˜NìbI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPвNíbI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP+›šHîbC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP¦Î’HïbC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPQ²JðbE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPŽ&Ћ;ñb6 loss/val_total_loss"trainer/global_step2:JP3É‘JòbE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP>Þ HóbC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÕa¾¡NôbI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPMW6NõbI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPC ÖkHöbC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP­‰ãH÷bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPv{JøbE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPpV4êHùbC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPMëëZNúbI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPžTÆJûbE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPˆ ìlHübC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP¼V%@ýb; particle/accuracy_2_of_2"trainer/global_step2:JPóUÉe@þb; particle/accuracy_3_of_5"trainer/global_step2:JP«³\NÿbI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÌë-‹N€bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPŠ!ÎÄJbE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP³'³…J‚bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP,ÁÌKNƒbI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP,ô™¼N„bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPrfÂH…bC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPŽ­‘zH†bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP`—I«N‡bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP0Ô0ÄHˆbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP‰^ñ+H‰bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPI„7ÀJŠbE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPüw`<‹b7 particle/sensitivity"trainer/global_step2:JPœÖxöJŒbE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPËh™HbC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP\¾£àJŽbE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPî õJbE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP dùüGbB validation_average_jet_accuracy"trainer/global_step2:JP'^JH‘bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPW{T7N’bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPOòH“bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP¶µéJ”bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPirtf@•b; particle/accuracy_2_of_3"trainer/global_step2:JP°ì¿$N–bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÃx3BJ—bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP~H¹¦H˜bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPꢣJ™bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPóÓÕHšbC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPÛCXJ›bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPö9ÙB;œb6 jet/accuracy_5_of_5"trainer/global_step2:JPàü¿g@b; particle/accuracy_5_of_5"trainer/global_step2:JP֧ǙHžbC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP@YþJŸbE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÎR´ŽN bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPß‚jJ¡bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP “2¿;¢b6 jet/accuracy_1_of_3"trainer/global_step2:JP¹×A»<£b7 particle/specificity"trainer/global_step2:JPiNGÒ;¤b6 jet/accuracy_2_of_5"trainer/global_step2:JP÷Iá@¥b; particle/accuracy_1_of_1"trainer/global_step2:JPk±/H¦bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPPUævÔN½bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP*U]:N¾bI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP@Ã8°J¿bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP¹{[HÀbC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÓs«|NÁbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÜ:aû@Âb; particle/accuracy_1_of_3"trainer/global_step2:JPM˜çJÃbE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP­$àhNÄbI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPé\v>HÅbC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP‡ÄëíNÆbI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPx8ëoNÇbI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP¯±*JÈbE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPLøk{HÉbC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP$±†JÊbE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPf¶1DJËbE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPXŠMNÌbI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPk™iHÍbC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPš£§XNÎbI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPðô·NÏbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPïß±JÐbE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP¨|ÌHÑbC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPTÌêtHÒbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÚÍ]NÓbI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPŒQZHÔbC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPžGó_NÕbI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP Y„oJÖbE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPñVOJJ×bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÝ̼×JØbE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP)‰HÙbC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP#GF0JÚbE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP3 njJÛbE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPnÓNdHÜbC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPâHظHÝbC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPJx”HÞbC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPŒ³ûHßbC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP¸ëqã9àb4 particle/accuracy"trainer/global_step2:JPúƒ{ÍHábC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPǶÁHJâbE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPý(JãbE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÙ.¦¸NäbI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPé¹vOHåbC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPÛŽ JæbE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP±„ÄtNçbI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP‘[öƒJèbE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPn ðJébE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPoÐ dJêbE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPJùbE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPâ·úÂHúbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPjÅépJûbE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP?]îèJübE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPbÃWNýbI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPâ¸jËHþbC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPT»1JÿbE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPq\Ží@€b; particle/accuracy_2_of_4"trainer/global_step2:JP(l„ŽHbC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPÈZœ$H‚bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPB§(‡HƒbC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPG1æúN„bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPsPLÊH…bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP_ª%íN†bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPa…úH‡bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPºLmûHˆbC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPÑ.ÏãJ‰bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP2Lš;;Šb6 jet/accuracy_1_of_1"trainer/global_step2:JPwÊóH‹bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPÞgÚèJŒbE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPX˜½JbE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPí)Þ@Žb; particle/accuracy_2_of_5"trainer/global_step2:JPòV½æJbE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÍ0àJbE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP7¤õÙH‘bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPÚ¬ûJ’bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP'ÖAùH“bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP:‰8H”bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP ¾ŠüJ•bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP™¯MN–bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP½CcqH—bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP‡ü‰{H˜bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP>ë"@™b; particle/accuracy_4_of_4"trainer/global_step2:JP•´@KHšbC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP<1¡J›bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPúßϪHœbC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPšB\ÀJbE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¥oz¥JžbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP±>mÝHŸbC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPÿ…!ÔJ bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP0F÷ÃN¡bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPï‘”xJ¢bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPáxÄ¡J£bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP+b5;¤b6 jet/accuracy_2_of_2"trainer/global_step2:JP¹[Dƒ@¥b; particle/accuracy_3_of_3"trainer/global_step2:JP‚ˆê;¦b6 jet/accuracy_3_of_5"trainer/global_step2:JPÿ;heH§bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPHe’7J¨bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP>IAeJ©bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPŸq7¸NªbI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÆ%FiJ«bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPõº6J¬bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPRGjcJ­bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPJ…íJ®bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÍøÁôJ¯bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP÷yS«N°bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPñŒkýJ±bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPúü/aH²bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP3_}ÁN³bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPcƒ J´bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP®ÆÄÌJµbE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP†4&J¶bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¿Öÿ„N·bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPéÂYH¸bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPWÒp³N¹bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPðt'JJºbE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPû©pJ»bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPE ÕN¼bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP¥ †J½bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP°ÃöUH¾bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP[Œ¿.N¿bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP1Œe!JÀbE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP–qtJÁbE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‰ÿÐNÂbI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP€k0 JÃbE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPQ™HÄbC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPî ‚¾JÅbE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPfÿFûÆbE "Purity/2FRt2›Ø g7SRqqt0FBt/event_purity"trainer/global_step2:JPWÞNÇbI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPC’>JÈbE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP”—JÉbE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÁìi­NÊbI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP#M¥ JËbE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPlë¾HÌbC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPÙ°‰‰0Í+ _wandbruntime‚355  _runtime‚355 =ªn1>Ϋ| 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9809842705726624 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.11158258467912674 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.06469827890396118 +jet/accuracy_2_of_4‚0.10998205840587616 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006349206669256091 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8042030334472656 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9885281324386597 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6210982203483582 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 particle/accuracy_1_of_3‚1 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.08039237558841705 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.008226446807384491 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.1953125 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15581953525543213 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6203703284263611 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9006571769714355 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.016509665176272392 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.09874359518289566 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.11696428060531616 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.02900644764304161 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0005952381179668009 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3461538553237915 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.015201796777546406 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.10261843353509903 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.7562732696533203 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.07083333283662796 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 _runtime‚ 355.200577006 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.5555555820465088 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05758531764149666 (particle/accuracy‚0.7668302655220032 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9602425694465637 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9828851819038391 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.11191636323928833 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.12834466993808746 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4444444477558136 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.05158518627285957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11845773458480835 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.963337779045105 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19514214992523193 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6203703284263611 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 *"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5 +jet/accuracy_1_of_4‚0.47828584909439087 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.11621323972940445 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.12816521525382996 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3461538553237915 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.06469298154115677 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5404305458068848 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.08194435387849808 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9790358543395996 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08563870191574097 particle/accuracy_2_of_4‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9687433838844299 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1564083844423294 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.009855953976511955 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9634806513786316 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1629026234149933 *jet/accuracy_1_of_1‚0.0791781097650528 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.14353741705417633 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0023467789869755507 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1 _step‚32 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.09457031637430191 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09198790788650513 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05758531764149666 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.017661361023783684 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9021015167236328 ;"Purity/1FRt*SRqqt1FBt/event_purity‚0.019078487530350685 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.0555555559694767 /particle/accuracy_4_of_4‚0.9502385258674622 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9764898419380188 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.24768519401550293 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.2040778547525406 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9813307523727417 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.07222919911146164 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7440041303634644 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.011970276944339275 +jet/accuracy_2_of_2‚0.04179459810256958 /particle/accuracy_3_of_3‚0.9980959296226501 jet/accuracy_3_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9572992920875549 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19505561888217926 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0040920088067650795 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.09923310577869415 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.15896503627300262 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.1433553844690323 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.0891970619559288 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.002801120514050126 *"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06322982907295227 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.11623724550008774 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.09332209080457687 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7570135593414307 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9831860065460205 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.16238512098789215 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.07870377600193024 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.029525717720389366 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.923031747341156 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0017607731278985739 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.10611334443092346 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.0891970619559288 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.0867430716753006 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1972692608833313 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.015816526487469673 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.3861549496650696 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.5801393389701843 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12264113128185272 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9502168297767639 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.40095946192741394 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_1_of_2‚0.3208785653114319 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4444444477558136 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.15952634811401367 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9804718494415283 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9447444677352905 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.018341032788157463 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18313397467136383 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9418367743492126 jet/accuracy_3_of_4‚0 'particle/f_score‚0.4971615970134735 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9831295013427734 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.09816509485244751 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.37485814094543457 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.05126480385661125 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1174282357096672 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9489546418190002 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.16317501664161682 jet/accuracy_4_of_5‚0 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0017414966132491827 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.6153846383094788 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9728480577468872 jet/accuracy_4_of_4‚0 /particle/accuracy_3_of_4‚0.9979296326637268 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 -jet/accuracy_3_of_3‚0.0047334483824670315 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.1364573985338211 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.1235567182302475 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.015192599967122078 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.05484943836927414 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9828851819038391 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.5740740895271301 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.06644753366708755 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3791482448577881 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.10530031472444534 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9767857789993286 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 *loss/val_total_loss‚37.842791591616795 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.39979979395866394 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0442708358168602 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12239423394203186 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.004122387617826462 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.15712396800518036 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.08606673032045364 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.012448256835341454 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3793420195579529 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.09372472763061523 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.05184224247932434 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.11115941405296326 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.1433553844690323 +particle/sensitivity‚0.5367465019226074 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.0690358430147171 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1860554814338684 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.16255666315555573 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0011614401591941714 7validation_average_jet_accuracy‚0.10959727317094803 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.884680449962616 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.015816526487469673 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 epoch‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.10365516692399979 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.13921013474464417 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚1 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30000001192092896 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0023952084593474865 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0012531328247860074 *jet/accuracy_1_of_3‚0.5067686438560486 +particle/specificity‚0.8297652006149292 jet/accuracy_2_of_5‚0 $ _timestamp‚1.784507181624517e+09 particle/accuracy_1_of_1‚1 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 trainer/global_step‚399 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9543823003768921 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.11289719492197037~Mÿ5>ϯ| =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0442708358168602 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12239423394203186 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.004122387617826462 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.15712396800518036 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.08606673032045364 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.012448256835341454 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3793420195579529 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.09372472763061523 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.05184224247932434 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.11115941405296326 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.1433553844690323 +particle/sensitivity‚0.5367465019226074 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.0690358430147171 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1860554814338684 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.16255666315555573 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0011614401591941714 7validation_average_jet_accuracy‚0.10959727317094803 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.884680449962616 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.015816526487469673 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 epoch‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.10365516692399979 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.13921013474464417 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚1 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30000001192092896 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0023952084593474865 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0012531328247860074 *jet/accuracy_1_of_3‚0.5067686438560486 +particle/specificity‚0.8297652006149292 jet/accuracy_2_of_5‚0 $ _timestamp‚1.784507181624517e+09 particle/accuracy_1_of_1‚1 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 trainer/global_step‚399 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9543823003768921 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.11289719492197037 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9809842705726624 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.11158258467912674 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.06469827890396118 +jet/accuracy_2_of_4‚0.10998205840587616 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006349206669256091 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8042030334472656 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9885281324386597 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6210982203483582 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 particle/accuracy_1_of_3‚1 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.08039237558841705 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.008226446807384491 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.1953125 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15581953525543213 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6203703284263611 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9006571769714355 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.016509665176272392 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.09874359518289566 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.11696428060531616 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.02900644764304161 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0005952381179668009 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3461538553237915 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.015201796777546406 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.10261843353509903 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.7562732696533203 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.07083333283662796 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 _runtime‚ 355.200577006 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.5555555820465088 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05758531764149666 (particle/accuracy‚0.7668302655220032 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9602425694465637 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9828851819038391 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.11191636323928833 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.12834466993808746 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4444444477558136 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.05158518627285957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11845773458480835 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.963337779045105 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19514214992523193 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6203703284263611 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 *"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5 +jet/accuracy_1_of_4‚0.47828584909439087 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.11621323972940445 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.12816521525382996 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3461538553237915 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.06469298154115677 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5404305458068848 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.08194435387849808 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9790358543395996 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08563870191574097 particle/accuracy_2_of_4‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9687433838844299 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1564083844423294 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.009855953976511955 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9634806513786316 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1629026234149933 *jet/accuracy_1_of_1‚0.0791781097650528 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.14353741705417633 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0023467789869755507 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1 _step‚32 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.09457031637430191 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09198790788650513 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05758531764149666 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.017661361023783684 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9021015167236328 ;"Purity/1FRt*SRqqt1FBt/event_purity‚0.019078487530350685 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.0555555559694767 /particle/accuracy_4_of_4‚0.9502385258674622 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9764898419380188 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.24768519401550293 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.2040778547525406 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9813307523727417 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.07222919911146164 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7440041303634644 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.011970276944339275 +jet/accuracy_2_of_2‚0.04179459810256958 /particle/accuracy_3_of_3‚0.9980959296226501 jet/accuracy_3_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9572992920875549 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19505561888217926 <"Purity/0FRt2SRqqt*FBt/event_purity‚0.0040920088067650795 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.09923310577869415 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.15896503627300262 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.1433553844690323 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.0891970619559288 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.002801120514050126 *"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06322982907295227 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.11623724550008774 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.09332209080457687 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7570135593414307 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9831860065460205 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.16238512098789215 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.07870377600193024 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.029525717720389366 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.923031747341156 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0017607731278985739 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.10611334443092346 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.0891970619559288 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.0867430716753006 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1972692608833313 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.015816526487469673 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.3861549496650696 particle/accuracy_1_of_4‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.5801393389701843 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12264113128185272 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9502168297767639 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.40095946192741394 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_1_of_2‚0.3208785653114319 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4444444477558136 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.15952634811401367 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9804718494415283 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9447444677352905 9 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.018341032788157463 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18313397467136383 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9418367743492126 jet/accuracy_3_of_4‚0 'particle/f_score‚0.4971615970134735 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9831295013427734 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.09816509485244751 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.37485814094543457 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.05126480385661125 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1174282357096672 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9489546418190002 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.16317501664161682 jet/accuracy_4_of_5‚0 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0017414966132491827 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.6153846383094788 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9728480577468872 jet/accuracy_4_of_4‚0 /particle/accuracy_3_of_4‚0.9979296326637268 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 -jet/accuracy_3_of_3‚0.0047334483824670315 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.1364573985338211 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.1235567182302475 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.015192599967122078 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.05484943836927414 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9828851819038391 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.5740740895271301 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.06644753366708755 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3791482448577881 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.10530031472444534 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9767857789993286 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 *loss/val_total_loss‚37.842791591616795 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.39979979395866394 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 µ'Ã$Ђ(: ²ÖõүʂÌ! disk.nvme0c0n1.out‚ 920.921875 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-1.out‚ 60.05859375 cpu‚25.64604702112162 network.recv‚137135) disk./.usagePercent‚76.22592455788323$ disk.nvme1n1.out‚217.21435546875 disk.md0.out‚ 15.870غ109375 disk.dm-0.in‚0' proc.memory.availableMB‚ 493.7890625 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 920.921875 disk.nvme1n1.in‚0" disk.dm-0.out‚1044.81298828125$ memory_percent‚96.98615074157715# proc.memory.rssMB‚ 2902.83203125* proc.memory.percent‚17.717480659484863 network.sent‚530948 disk.rbd0.in‚0& disk.nvme1c1n1.out‚217.21435546875 disk.dm-1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1170.2585067749023 disk.rbd0.out‚0b'ÿ²Ñ‚(:’ ²ÖõÒ«©¾Ì gpu.0.gpu‚88.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚272.45' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.81666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁTo/Ò‚(:¤ ÁÖõÒò¿áË network.sent‚576355 network.recv‚141649$ disk./.usageGB‚1170.321491241455$ disk.nvme1n1.out‚709.03662109375 disk.dm-1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.717480659484863 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚954.10888671875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚0 disk.dm-1.out‚ 551.23828125) disk./.usagePercent‚76.23002711229148 disk.nvme0n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 16.390625 disk.dm-0.out‚1077.537109375 disk.rbd0.out‚0' proc.memory.availableMB‚ 493.7578125# proc.memory.rssMB‚ 2902.83203125$ disk.nvme0n1.out‚954.10888671875& disk.nvme1c1n1.out‚709.03662109375$ memory_percent‚96.98634147644043 cpu‚25.639843462916115 proc.cpu.threads‚17PžÓ‚(:“ ÁÖõÒêè Ì gpu.0.gpu‚91.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚278.941' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.98033333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#àoÙ.Ô‚(:£ ÐÖõÒ°¸Ë% disk./.usageGB‚1170.3213996887207 disk.nvme1n1.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚709.76123046875( disk./.usagePercent‚76.2300211489152 disk.md0.in‚0 disk.md0.out‚ 16.64453125 disk.dm-0.out‚ 1088.25390625 disk.dm-1.in‚0 disk.dm-1.out‚ 551.28125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0$ memory_percent‚96.98636531829834# proc.memory.rssMB‚ 2902.83203125* proc.memory.percent‚17.717480659484863 cpu‚25.635338050053388 proc.cpu.threads‚17$ disk.nvme1n1.out‚709.76123046875& disk.nvme0c0n1.out‚964.69482421875 disk.nvme0n1.in‚0$ disk.nvme0n1.out‚964.69482421875( proc.memory.availableMB‚ 493.75390625 network.sent‚579816 network.recv‚145341OY=NžÕ‚(:“ ÐÖõÒ£×ŽÌ gpu.0.gpu‚56.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚236.996' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚78.99866666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ÍØ0Ö+ _wandbruntime‚398  _runtime‚398  eÃ×½ & lr-AdamW/pg1‚0.0006144799457263714 & lr-AdamW/pg2‚0.0006144799457263714 trainer/global_step‚449 % _timestamp‚1.7845072247547843e+09 _runtime‚ 398.33037517 _step‚33nãà8ÇØÁ _step‚33 & lr-AdamW/pg1‚0.0006144799457263714 & lr-AdamW/pg2‚0.0006144799457263714 trainer/global_step‚449 % _timestamp‚1.7845072247547843e+09 _runtime‚ 398.33037517!|¬ à0Ù+ _wandbruntime‚398  _runtime‚398`…í¹'Ú³O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92687360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4440573 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68701444888064 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 521179395072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111299584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4450714 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68300619505200 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3633563 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚34 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3633561 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68701461927424 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4440573 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 % _timestamp‚1.7845072247557945e+09 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69222624283136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15081930170880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1874610 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4450714 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1874335 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68821352246433 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1084578 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69222982745600 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 520392179967 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3633563 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 520732741233 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8074134 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4450714 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1874335 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68821352246433 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1084578 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚69222982745600 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 520392179967 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3633563 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 520732741233 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8074134 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4440573 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 % _timestamp‚1.7845072247557945e+09 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69222624283136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15081930170880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1874610 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚ 398.943249244 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68849719390208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1086827 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3641354 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4460384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4450229 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚791766 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚449 _step‚35 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3641354 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 522630924288 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68849719390208 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3641356 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8101740 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68969533701531 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 522289494016 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68448050172672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8101740 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3641356 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 521841129597 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8091583 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 522289494016 $ _timestamp‚1.784507225367503e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1086554 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15641226405888 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68849736429568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 521500568219 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚791763 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15641110919168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1878317 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 526000478208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 526097352704 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69372367353856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4460384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 SEDeviceStatsMonitor.on_train_batcÝýîh_end/requested_bytes.small_pool.peak‚ 560516791 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚69372367353856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 522630924288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15115129053184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68969891302269 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68448033133312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚ 398.943249244 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68849719390208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1086827 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3641354 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4460384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4450229 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚791766 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚449 _step‚35 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3641354 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 522630924288 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68849719390208 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3641356 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8101740 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68969533701531 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 522289494016 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68448050172672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8101740 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3641356 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 521841129597 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8091583 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 522289494016 $ _timestamp‚1.784507225367503e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1086554 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15641226405888 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68849736429568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 521500568219 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚791763 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15641110919168 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1878317 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 526000478208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 526097352704 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚69372367353856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4460384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚69372367353856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 522630924288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15115129053184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68969891302269 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68448033133312 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E5fÃáû 0loss/FBt2/assignment_loss‚0.3631292010113905 /loss/FBt2/detection_loss‚3.0129231851427787 %loss/total_loss‚20.85617975214892 epoch‚1 $ _timestamp‚1.784507225368845e+09 /loss/FRt1/assignment_loss‚3.858352602818969 .loss/FBt1/detection_loss‚2.290850298304477 _runtime‚ 398.944230806 .loss/FRt2/assignment_loss‚3.08828678816916 2loss/SRqqt2/assignment_loss‚1.4653236876067974 /loss/FRt1/detection_loss‚0.6235621319370416 /loss/FRt2/detection_loss‚1.3846690530470762 trainer/global_step‚449 _step‚36 2loss/SRqqt1/assignment_loss‚1.4983604677622737 1loss/FBt1/assignment_loss‚0.13025457418131833 0loss/SRqqt1/detection_loss‚2.404352533949327 1loss/SRqqt2/detection_loss‚0.7361152282183113$LÇw0â‚(:¥ ßÖõÒ¡ŸÂË% disk.nvme1c1n1.out‚710.4189453125$ memory_percent‚96.98679447174072# proc.memory.rssMB‚ 2902.83203125* proc.memory.percent‚17.717480659484863 network.sent‚598842% disk./.usageGB‚1170.3215789794922 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 17.0390625 disk.dm-1.in‚0! disk.dm-1.out‚551.41845703125$ disk.nvme0n1.out‚974.57080078125# disk.nvme1n1.out‚710.4189453125 disk.dm-0.in‚0 disk.dm-0.out‚ 1097.796875 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚974.57080078125( proc.memory.availableMB‚ 493.68359375 disk.nvme0n1.in‚0 disk.nvme0c0n1.in‚0 cpu‚25.629914884663595 proc.cpu.threads‚17 network.recv‚149515) disk./.usagePercent‚76.23003282719377ö8Œžœã‚(:‘ ßÖõÒ·”…Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.551' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.85033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"þ£.ä‚(:£ îÖõÒŠØ°Ë proc.cpu.threads‚17 network.recv‚154060$ disk./.usageGB‚1170.321491241455 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚551.47705078125# memory_percent‚96.9872236251831* proc.memory.percent‚17.717480659484863 network.sent‚647508) disk./.usagePercent‚76.23002711229148% disk.nvme1c1n1.out‚711.0400390625 disk.md0.in‚0 disk.dm-0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 493.61328125# proc.memory.rssMB‚ 2902.83203125" disk.dm-0.out‚1123.19482421875# disk.nvme0c0n1.out‚ 1000.1328125! disk.nvme0n1.out‚ 1000.1328125 disk.nvme1n1.in‚0# disk.nvme1n1.out‚711.0400390625 disk.md0.out‚ 17.37109375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 cpu‚25.624318174406287§¯Æžå‚(:“ îÖõÒ©ØƒÌ gpu.0.gpu‚46.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.843' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.94766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›è{/æ‚(:¤ ýÖõÒ·Ý­Ë disk.md0.in‚0# proc.memory.rssMB‚ 2902.83203125 proc.cpu.threads‚17 disk.nvme1n1.in‚0" disk.dm-0.out‚1134.18701171875$ memory_percent‚96.98808193206787( proc.memory.availableMB‚ 493.47265625* proc.memory.percent‚17.717480659484863 cpu‚25.62018822520076) disk./.usagePercent‚76.23819967101245% disk./.usageGB‚1170.4469604492188 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0% disk.nvme0c0n1.out‚1011.373046875# disk.nvme0n1.out‚1011.373046875 disk.nvme1c1n1.in‚0# disk.nvme1c1n1.out‚ 712.14453125 network.sent‚653729 network.recv‚158188! disk.dm-1.out‚551.63037109375 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme1n1.out‚ 712.14453125 disk.md0.out‚ 17.92578125 disk.dm-0.in‚0 disk.nvme0n1.in‚0Ãâ³–žç‚(:“ ýÖõÒ‡è†Ì gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚278.592' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.86399999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬!‚0è+ _wandbruntime‚440  _runtime‚440:ÄÕûÄé¾ _step‚37 & lr-AdamW/pg1‚0.0006144507403776173 & lr-AdamW/pg2‚0.0006144507403776173 trainer/global_step‚499 % _timestamp‚1.7845072667147875e+09 _runtime‚ 440.292045421>BÇÈê & lr-AdamW/pg1‚0.0006144507403776173 & lr-AdamW/pg2‚0.0006144507403776173 trainer/global_step‚499 % _timestamp‚1.7845072667147875e+09 _runtime‚ 440.292045421 _step‚37%<‘90ë+ _wandbruntime‚440  _runtime‚440#átuº'ì´O K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4925225 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17321847963648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1199114 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17321721984000 _runtime‚ 440.292108821 _step‚38 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚876732 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8948436 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76248368833441 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 577089920000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4023211 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1199384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76691918492160 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 576748496896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76115187034624 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚876729 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16740943674880 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 580778309120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76115169995264 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76692276954624 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76692276954624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16740962287104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107367424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 577089920000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4023213 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125979648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 575875393375 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76115169995264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76691918492160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4935366 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 580885676544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8958579 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 576215954641 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4925225 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2076116 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76115187034624 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 576748496896 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4935366 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2075843 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76115187034624 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚876729 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16740943674880 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 580778309120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76115169995264 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76692276954624 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76692276954624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16740962287104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107367424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 577089920000 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4023213 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125979648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 575875393375 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76115169995264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚i‚y"1180 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76691918492160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4935366 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 580885676544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8958579 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 576215954641 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4925225 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2076116 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76115187034624 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 576748496896 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4935366 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2075843 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚878463 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 577858595840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76263461536768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚39 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4031006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 577324343005 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75819566506912 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4945036 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚878466 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76841661562880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2079777 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 581893774848 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4945036 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 440.903812403 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4031004 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8976042 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17356017719808 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8965885 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8976042 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4945036 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚878466 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76841661562880 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2079777 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 581893774848 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4945036 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 440.903812403 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4031004 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8976042 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚17356017719808 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8965885 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8976042 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚878463 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 577858595840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚76263461536768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚39 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4031006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 577324343005 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75819566506912 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84161571163648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 632659021824 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9832881 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 632659021824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚549 _runtime‚ 482.512064775 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19036609901056 % _timestamp‚1.7845073089339333e+09 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚963429 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚83528912141824 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 636749603840 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4412863 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84161212701184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1313504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _step‚42 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18399957178880 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83043686252400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5409877 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84161212701184 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83528895102464 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2276648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 636652722176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 631358606783 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18399975791104 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19036725394944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 631699168049†˜×½'ÿ·O RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚963429 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚83528912141824 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 636749603840 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4412863 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84161212701184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1313504 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _step‚42 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18399957178880 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83043686252400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5409877 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84161212701184 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83528895102464 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2276648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 636652722176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 631358606783 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18399975791104 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19036725394944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 631699168049 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 632317598720 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5420018 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84161571163648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 632659021824 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9832881 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 632659021824 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚549 _runtime‚ 482.512064775 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19036609901056 % _timestamp‚1.7845073089339333e+09*9‰k|0€+ _wandbruntime‚483  _runtime‚483>šSÂ&¼M L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9850344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83677169604608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117583872 _runtime‚ 483.116734151 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1315478 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 637769668608 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4420656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 633427697664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83677186643968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4420654 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18433137448960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83823924476285 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98971648 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 632466995035 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 $ _timestamp‚1.784507309540911e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚965166 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2280641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9850344 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4420656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 633769127936 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 637868640256 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19070907117568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 632807556413 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚549 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84310955771904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9840187 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5429688 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚965163 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 633427697664 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5419533 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5429688 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4420654 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1315747 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18433156061184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83191116919872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232êBô—‚ÀM I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4420654 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18433137448960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83823924476285 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98971648 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 632466995035 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 $ _timestamp‚1.784507309540911e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚965166 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2280641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9850344 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4420656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 633769127936 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 637868640256 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19070907117568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 632807556413 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚549 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytevæNG/s.all.allocated‚84310955771904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9840187 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5429688 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚965163 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 633427697664 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5419533 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5429688 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4420654 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1315747 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18433156061184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83191116919872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9850344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83677169604608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117583872 _runtime‚ 483.116734151 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1315478 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 637769668608 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4420656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 633427697664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83677186643968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372+–”àÇ0ƒ+ _wandbruntime‚483  _runtime‚483æ%½„û 0loss/FRt1/assignment_loss‚3.8283801427769717 1loss/SRqqt1/detection_loss‚2.1205283363923333 epoch‚1 _step‚44 0loss/FRt2/assignment_loss‚2.3156733829903122 2loss/SRqqt1/assignment_loss‚1.4237744806470065 0loss/FBt1/assignment_loss‚0.1281788450486308 /loss/FRt1/detection_loss‚0.6415593736086355 /loss/FBt1/detection_loss‚1.1711001206787524 /loss/FBt2/detection_loss‚2.1871193516254426 trainer/global_step‚549 2loss/SRqqt2/assignment_loss‚1.3123896643384751 0loss/FBt2/assignment_loss‚0.2476341940308339 .loss/FRt2/detection_loss‚1.423405397877147 &loss/total_loss‚17.548088295350613 _runtime‚ 483.12111519 1loss/SRqqt2/detection_loss‚0.7483450053360717 % _timestamp‚1.7845073095457506e+09îY$ …ÿ 0loss/FRt1/assignment_loss‚3.8283801427769717 1loss/SRqqt1/detection_loss‚2.1205283363923333 epoch‚1 _step‚44 0loss/FRt2/assignment_loss‚2.3156733829903122 2loss/SRqqt1/assignment_loss‚1.4237744806470065 0loss/FBt1/assignment_loss‚0.1281788450486308 /loss/FRt1/detection_loss‚0.6415593736086355 /loss/FBt1/detection_loss‚1.1711001206787524 /loss/FBt2/detection_loss‚2.1871193516254426 trainer/global_step‚549 2loss/SRqqt2/assignment_loss‚1.3123896643384751 0loss/FBt2/assignment_loss‚0.2476341940308339 .loss/FRt2/detection_loss‚1.423405397877147 &loss/total_loss‚17.548088295350613 _runtime‚ 483.12111519 1loss/SRqqt2/detection_loss‚0.7483450053360717 % _timestamp‚1.7845073095457506e+09, Œ¼Ê7†‚(:¬ ¹×õÒëÒ™Ë% disk.nvme0n1.out‚1089.97607421875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚715.65869140625 network.sent‚730956! disk.dm-1.out‚551.95947265625' disk.nvme0c0n1.out‚1089.97607421875& disk.nvme1c1n1.out‚715.65869140625 disk.md0.in‚0 disk.md0.out‚ 20.1171875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0' proc.memory.availableMB‚ 492.4453125* proc.memory.percent‚17.717504501342773 network.recv‚173407% disk./.usageGB‚1170.2089500427246 disk.dm-0.in‚0" disk.dm-0.out‚1211.28466796875$ memory_percent‚96.99435234069824" proc.memory.rssMB‚ 2902.8359375 cpu‚25.604631386384927) disk./.usagePercent‚76.22269663199639 disk.nvme0n1.in‚05„&“‡‚(:ˆ ¹×õÒ‘ã‰Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚223.518' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚74.506 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÌ®³,ˆ‚(:¡ È×õÒÛ˜¬Ë network.recv‚177580) disk./.usagePercent‚76.22270160147663 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0% disk.nvme0n1.out‚1102.42041015625 disk.nvme1c1n1.in‚0! disk.nvme1c1n1.out‚ 716.328125# disk./.usageGB‚1170.20902633667 disk.nvme1n1.in‚0 disk.md0.out‚ 20.47265625! disk.dm-0.out‚1223.5791015625( proc.memory.availableMB‚ 492.41015625 cpu‚25.601158982252773 network.sent‚794304 disk.dm-1.in‚0! disk.dm-1.out‚551.99462890625$ memory_percent‚96.99456691741943" proc.memory.rssMB‚ 2902.8359375* proc.memory.percent‚17.717504501342773 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚1102.42041015625 disk.nvme1n1.out‚ 716.328125 disk.md0.in‚0 disk.dm-0.in‚0¸O¨ž‰‚(:“ È×õÒÆéË gpu.0.gpu‚72.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚279.772' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.25733333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬w ´0Š+ _wandbruntime‚524  _runtime‚524ŠÖ9a˽ & lr-AdamW/pg1‚0.0006143331390155881 & lr-AdamW/pg2‚0.0006143331390155881 trainer/global_step‚599 $ _timestamp‚1.784507350745528e+09 _runtime‚ 524.321577514 _step‚450^(]ÇŒÁ & lr-AdamW/pg2‚0.0006143331390155881 trainer/global_step‚599 $ _timestamp‚1.784507350745528e+09 _runtime‚ 524.321577514 _step‚45 & lr-AdamW/pg1‚0.0006143331390155881-ÖAq0+ _wandbruntime‚524  _runtime‚524dŒ2òÀ'ŽºO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5894529 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2478057 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20058989295104 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 688228123648 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4802511 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20058970682880 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚91630506910208 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1050132 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20751574167552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20751443993600 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90415219626000 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10697040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91630865372672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 trainer/global_step‚599 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91630506910208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91630865372672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1427925 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 692584872448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111561728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 _runtime‚ 524.321807274 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5904670 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90942620209664 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91630865372672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 trainer/global_step‚599 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91630506910208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91630865372672 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1427925 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 692584872448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111561728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 _runtime‚ 524.321807274 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5904670 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90942620209664 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5894529 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2478057 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20058989295104 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 688228123648 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4802511 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20058970682880 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚91630506910208 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1050132 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452.ÙŸöº0+ _wandbruntime‚524  _runtime‚5244µ¢ù‘ÃM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90562650293472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 % _timestamp‚1.7845073513564734e+09 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4810306 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91090911751168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 693699014144 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 688290769821 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107360256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125972480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 687950208443 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91090911751168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91250941063293 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5914340 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2481736 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91780249980928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5914340 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 689338229760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1051866 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10724646 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10714489 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20785868579328ïÑ€Í&’ÇM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5914340 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2481736 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91780249980928 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5914340 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 689338229760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1051866 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10724646 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10714489 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20785868579328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90562650293472 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 % _timestamp‚1.7845073513564734e+09 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4810306 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91090911751168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 693699014144 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 688290769821 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107360256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125972480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 687950208443 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91090911751168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91250941063293/v‚}0“+ _wandbruntime‚524  _runtime‚524ËK§~”ü _step‚48 2loss/SRqqt2/assignment_loss‚1.2195364047508446 1loss/FBt2/assignment_loss‚0.15465202142814227 &loss/total_loss‚15.688753774462127 1loss/SRqqt1/assignment_loss‚1.288970766419714 1loss/FBt1/assignment_loss‚0.12788560644320926 /loss/FRt1/detection_loss‚0.6011420759831043 1loss/SRqqt2/detection_loss‚0.6880864302515941 epoch‚1 % _timestamp‚1.7845073513592505e+09 _runtime‚ 524.934660945 /loss/FRt2/assignment_loss‚2.082680213196077 1loss/SRqqt1/detection_loss‚1.9978092870154938 /loss/FBt1/detection_loss‚0.9358031869334974 trainer/global_step‚599 /loss/FRt1/assignment_loss‚3.698284108187402 /loss/FRt2/detection_loss‚1.4596975553562923 /loss/FBt2/detection_loss‚1.4342061184967558£… •€ trainer/global_step‚599 /loss/FRt1/assignment_loss‚3.698284108187402 /loss/FRt2/detection_loss‚1.4596975553562923 /loss/FBt2/detection_loss‚1.4342061184967558 _step‚48 2loss/SRqqt2/assignment_loss‚1.2195364047508446 1loss/FBt2/assignment_loss‚0.15465202142814227 &loss/total_loss‚15.688753774462127 1loss/SRqqt1/assignment_loss‚1.288970766419714 1loss/FBt1/assignment_loss‚0.12788560644320926 /loss/FRt1/detection_loss‚0.6011420759831043 1loss/SRqqt2/detection_loss‚0.6880864302515941 epoch‚1 % _timestamp‚1.7845073513592505e+09 _runtime‚ 524.934660945 /loss/FRt2/assignment_loss‚2.082680213196077 1loss/SRqqt1/detection_loss‚1.9978092870154938 /loss/FBt1/detection_loss‚0.93580318693349740&W70–‚(:¥ ××õÒ¨íÃË network.sent‚797765 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚1131.1630859375$ disk.nvme1c1n1.out‚ 717.033203125* proc.memory.percent‚17.717504501342773 cpu‚25.59510341617776% disk./.usageGB‚1169.9449424743652 disk.rbd0.in‚0$ disk.nvme0n1.out‚1131.1630859375 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 717.033203125 disk.md0.in‚0 disk.md0.out‚ 20.89453125" disk.dm-0.out‚1251.99951171875 disk.rbd0.out‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.dm-0.in‚0 disk.dm-1.in‚0$ memory_percent‚99.95768070220947& proc.memory.availableMB‚ 6.93359375" proc.memory.rssMB‚ 2902.8359375 network.recv‚181576) disk./.usagePercent‚76.20550024259015! disk.dm-1.out‚552.11962890625 proc.cpu.threads‚17TQñž—‚(:“ ××õÒ–Ò®Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.845' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.61500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fL±ñ/˜‚(:¤ æ×õÒåŽÚË$ disk.nvme1c1n1.out‚ 721.169921875 disk.md0.out‚ 21.0859375" disk.dm-0.out‚1269.54248046875& disk.nvme0c0n1.out‚1146.1201171875$ disk.nvme0n1.out‚1146.1201171875 disk.md0.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚553.24072265625 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚99.9565839767456 disk.dm-0.in‚0& proc.memory.availableMB‚ 7.11328125* proc.memory.percent‚17.717504501342773) disk./.usagePercent‚76.20550024259015 disk.nvme0c0n1.in‚0 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 721.169921875" proc.memory.rssMB‚ 2902.8359375 cpu‚25.592072359000365 proc.cpu.threads‚17 network.sent‚862175 network.recv‚186028% disk./.usageGB‚1169.9449424743652 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0‚}^ž™‚(:“ æ×õÒÀœ¢Ì gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.908' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.63600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wv’0š‚(:¥ õ×õÒžŽ·Ë disk.nvme1c1n1.in‚0 disk.dm-0.in‚0" disk.dm-0.out‚1270.47607421875% disk./.usageGB‚1169.9443740844727 disk.rbd0.out‚0 disk.nvme0n1.in‚0 disk.md0.out‚ 21.54296875 disk.dm-1.out‚ 562.69921875 proc.cpu.threads‚17 disk.rbd0.in‚0$ disk.nvme0n1.out‚1147.4306640625 disk.nvme1n1.in‚0 disk.dm-1.in‚0$ memory_percent‚99.95696544647217* proc.memory.percent‚17.717504501342773 cpu‚25.58836738022266 disk.nvme0c0n1.in‚0& disk.nvme1c1n1.out‚731.23193359375 disk.md0.in‚0& proc.memory.availableMB‚ 7.05078125" proc.memory.rssMB‚ 2902.8359375$ disk.nvme1n1.out‚731.23193359375 network.sent‚867329 network.recv‚189870( disk./.usagePercent‚76.2054632199624& disk.nvme0c0n1.out‚1147.4306640625ÍÉО›‚(:“ õ×õÒ‡¢Ì gpu.0.gpu‚93.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.597' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.19899999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ªr‰0œ+ _wandbruntime‚566  _runtime‚566üê¤øľ _runtime‚ 566.191325347 _step‚49 & lr-AdamW/pg1‚0.0006142447505556286 & lr-AdamW/pg2‚0.0006142447505556286 trainer/global_step‚649 % _timestamp‚1.7845073926156707e+09pê—øȞ & lr-AdamW/pg1‚0.0006142447505556286 & lr-AdamW/pg2‚0.0006142447505556286 trainer/global_step‚649 % _timestamp‚1.7845073926156707e+09 _runtime‚ 566.191325347 _step‚491‰·ù20Ÿ+ _wandbruntime‚566  _runtime‚566 ó¨f$ ¹O OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99100159581696 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1136832 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1542165 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21717984186880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 743455802368 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 742325033599 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5192163 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98356345316864 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22466438641152 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97786735960240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 748435842048 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚98529418594465 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5192161 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1541869 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21718002799104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6389322 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2678698 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101075968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 743797225472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6379181 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6389322 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98356362356224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99099801119232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98356345316864 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11581485 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2678997 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5192161 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 743797225472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98356362356224 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99099801119232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_po\™z…Yol.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5192163 _step‚50 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 743455802368 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 742325033599 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5192163 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98356345316864 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22466438641152 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97786735960240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 748435842048 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚98529418594465 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5192161 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1541869 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21718002799104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6389322 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2678698 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101075968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 743797225472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6379181 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6389322 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98356362356224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99099801119232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98356345316864 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11581485 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2678997 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5192161 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 743797225472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98356362356224 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99099801119232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5192163 _step‚50 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BŒÉ&£ÃM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6398992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98504636858368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1544113 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2682959 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98504636858368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98504619819008 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6388837 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 743433421851 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99249185720320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 744565901312 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11598948 _step‚51 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132263936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113651712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚ 566.800494959 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚649 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6398992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98677957650301 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22500741885952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21751183069184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5199956 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11598948 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1138563 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1544393 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2682676 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 744907331584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚98677600049563 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 744907331584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22500609622016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11588791 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99249544189952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 749445165056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 744565901312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21751164456960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2uÐÍ&¤ÇM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22500741885952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21751183069184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5199956 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11598948 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1138563 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1544393 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2682676 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 744907331584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚98677600049563 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 744907331584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22500609622016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11588791 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99249544189952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 749445165056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 744565901312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21751164456960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6398992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98504636858368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1544113 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2682959 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98504636858368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98504619819008 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6388837 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 743433421851 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99249185720320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 744565901312 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11598948 _step‚51 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132263936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113651712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚ 566.800494959 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚649 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6398992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚986779576503013Ö O~0¥+ _wandbruntime‚566  _runtime‚566áž“K¦þ 1loss/SRqqt2/detection_loss‚0.7232346133602766 1loss/FBt2/assignment_loss‚0.15313047904029986 /loss/FRt1/detection_loss‚0.6484694299290771 2loss/SRqqt2/assignment_loss‚1.1400284693248708 1loss/SRqqt1/detection_loss‚1.8112083370797336 /loss/FBt1/detection_loss‚2.0464894415257486 trainer/global_step‚649 _runtime‚ 566.804404496 2loss/SRqqt1/assignment_loss‚1.1895228162058629 1loss/FBt1/assignment_loss‚0.12019098446370681 /loss/FRt2/detection_loss‚1.5461552414234248 /loss/FBt2/detection_loss‚2.6085204215099416 %loss/total_loss‚17.53238807502901 epoch‚1 % _timestamp‚1.7845073932289994e+09 _step‚52 0loss/FRt1/assignment_loss‚3.5143900552015195 0loss/FRt2/assignment_loss‚2.0310477859645495‘!K˜§‚ 2loss/SRqqt2/assignment_loss‚1.1400284693248708 1loss/SRqqt1/detection_loss‚1.8112083370797336 /loss/FBt1/detection_loss‚2.0464894415257486 trainer/global_step‚649 _runtime‚ 566.804404496 2loss/SRqqt1/assignment_loss‚1.1895228162058629 1loss/FBt1/assignment_loss‚0.12019098446370681 /loss/FRt2/detection_loss‚1.5461552414234248 /loss/FBt2/detection_loss‚2.6085204215099416 %loss/total_loss‚17.53238807502901 epoch‚1 % _timestamp‚1.7845073932289994e+09 _step‚52 0loss/FRt1/assignment_loss‚3.5143900552015195 0loss/FRt2/assignment_loss‚2.0310477859645495 1loss/SRqqt2/detection_loss‚0.7232346133602766 1loss/FBt2/assignment_loss‚0.15313047904029986 /loss/FRt1/detection_loss‚0.64846942992907714o(Œ£¨‚(:¡ „ØõÒ¥•ÍË% disk./.usageGB‚1170.3731117248535& disk.nvme0c0n1.out‚1210.1904296875 disk.nvme0n1.in‚0& disk.nvme1c1n1.out‚731.9252929J'¸J‰6875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0$ disk.nvme0n1.out‚1210.1904296875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.717504501342773 disk.dm-0.in‚0" disk.dm-0.out‚1333.15576171875 disk.dm-1.in‚0" proc.memory.rssMB‚ 2902.8359375 cpu‚25.585664067208945 disk.md0.out‚ 21.859375 network.recv‚193387) disk./.usagePercent‚76.23338946261869$ disk.nvme1n1.out‚731.92529296875 disk.dm-1.out‚ 562.78125$ memory_percent‚99.95884895324707% proc.memory.availableMB‚ 6.7421875 proc.cpu.threads‚17 network.sent‚883490ì,°ž©‚(:“ „ØõÒË™–Ì gpu.0.gpu‚82.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚269.736' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.91199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aýï*ª‚(:Ÿ “ØõÒ‹”ÙË) disk./.usagePercent‚76.24064117665206% disk.nvme0c0n1.out‚1356.126953125 disk.nvme0n1.in‚0 disk.md0.in‚0 disk.md0.out‚22.65625# memory_percent‚99.9608039855957 proc.cpu.threads‚17 disk.nvme1n1.in‚0% disk./.usageGB‚1170.4844436645508 disk.rbd0.in‚0% disk.nvme1c1n1.out‚760.8349609375" disk.dm-0.out‚1505.53466796875! disk.dm-1.out‚563.57080078125$ proc.memory.availableMB‚6.421875" proc.memory.rssMB‚ 2902.8359375 disk.rbd0.out‚0# disk.nvme1n1.out‚760.8349609375 disk.dm-1.in‚0* proc.memory.percent‚17.717504501342773 cpu‚25.58227095092891 network.sent‚933470 network.recv‚197830 disk.nvme0c0n1.in‚0# disk.nvme0n1.out‚1356.126953125 disk.nvme1c1n1.in‚0 disk.dm-0.in‚0'j‰‘«‚(:† “ØõÒÅÞÙË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.117' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.039 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jOb+%¬‚(:š ¢ØõÒöµ·Ë* proc.memory.percent‚17.717504501342773 proc.cpu.threads‚17 network.sent‚941125$ disk./.usageGB‚1170.420021057129 disk.nvme1n1.in‚0 disk.rbd0.out‚0 cpu‚25.57866832195954 network.recv‚202711) disk./.usagePercent‚76.23644494754137 disk.nvme0c0n1.in‚0# disk.nvme1c1n1.out‚ 847.20703125 disk.md0.out‚ 23.18359375 disk.dm-0.in‚0" disk.dm-0.out‚1765.45751953125" proc.memory.rssMB‚ 2902.8359375 disk.nvme0n1.in‚0 disk.nvme0n1.out‚ 1530.859375 disk.nvme1c1n1.in‚0! disk.nvme1n1.out‚ 847.20703125 disk.md0.in‚0! disk.dm-1.out‚563.63330078125 disk.rbd0.in‚0& proc.memory.availableMB‚ 6.66015625" disk.nvme0c0n1.out‚ 1530.859375 disk.dm-1.in‚0$ memory_percent‚99.95934963226318Hš£<“­‚(:ˆ ¢ØõÒª™ûË gpu.0.gpu‚57.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.754' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«z à0®+ _wandbruntime‚608  _runtime‚608‘ªcį¾ _runtime‚ 608.401695936 _step‚53 & lr-AdamW/pg1‚0.0006141366440568009 & lr-AdamW/pg2‚0.0006141366440568009 trainer/global_step‚699 % _timestamp‚1.7845074348254416e+09ï_0PÈ°Â trainer/global_step‚699 % _timestamp‚1.7845074348254416e+09 _runtime‚ 608.401695936 _step‚53 & lr-AdamW/pg1‚0.0006141366440568009 & lr-AdamW/pg2‚0.00061413664405680095n²ù0±+ _wandbruntime‚608  _runtime‚608AJÌ'²ÆO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1656336 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7845074348261466e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23376997690880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6863833 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106569095328256 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 799024904192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚106569453790720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105770087463424 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 804136281088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105158269333840 _runtime‚ 608.401762686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24181133971968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 799366327296 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132271104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105770070424064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ÅU”ŽÐ'³ÊO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105770087463424 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 804136281088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105158269333840 _runtime‚ 608.401762686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24181133971968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 799366327296 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132271104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105770070424064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1656336 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7845074348261466e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23376997690880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6863833 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106569095328256 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 799024904192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚106569453790720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚06"N0´+ _wandbruntime‚609  _runtime‚609.ÚÎL µÐM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105918361965568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5589606 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2883561 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23410177960960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚55 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6873489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚106718838398976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1225263 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚106718838398976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128069632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106104974237309 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5589606 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5589604 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12463093 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5589604 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1658569 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1658298 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105918361965568 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 800476433408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12463093 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 800135003136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 805362226176 % _timestamp‚1.7845074354349535e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚105305700001312 trainer/global_step‚699 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6873489 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 800135003136 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch·ý˜wŠ_end/segment.small_pool.allocated‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _runtime‚ 609.010740919 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6883644 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1225266 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24215430729728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24215558799360 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 805252768768 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12473250 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799257196637:¸èÚ&¶ÔM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚106718479929344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 800476433408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12463093 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 800135003136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 805362226176 % _timestamp‚1.7845074354349535e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚105305700001312 trainer/global_step‚699 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6873489 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 800135003136 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _runtime‚ 609.010740919 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6883644 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1225266 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24215430729728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24215558799360 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 805252768768 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12473250 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799257196637 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105918361965568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5589606 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2883561 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23410177960960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚55 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6873489 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚106718838398976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1225263 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚106718838398976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128069632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106104974237309 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5589606 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5589604 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12463093 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5589604 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1658569 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1658298 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105918361965568 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1310232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13330089 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚114038389537280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113383451768481 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113383094167855 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7358626 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 _step‚58 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13319946 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1770651 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25896050691584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 854594006016 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5971461 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚114038747999744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845074768651795e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5971463 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112529802707440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 _runtime‚ 650.440638841 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1770367 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5971461 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132271104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25036029807104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 860020884480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113183812570624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25895918420480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112529819746800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113183795531264 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 859907225600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 853632021681 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114038389537280 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 854935429120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚114038389537280 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 854935429120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1310232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 150669312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13330089 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚114038389537280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113383451768481 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113383094167855 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425—9°oå344 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7358626 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613312 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 _step‚58 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13319946 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1770651 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25896050691584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 854594006016 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5971461 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚114038747999744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845074768651795e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5971463 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112529802707440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423104 _runtime‚ 650.440638841 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1770367 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5971461 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132271104 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25036029807104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 860020884480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113183812570624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25895918420480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600626 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112529819746800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113183795531264 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 859907225600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 853632021681:\ÚŸ*0Æ+ _wandbruntime‚651  _runtime‚6515• !Ö&ÇÐM MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7368296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132263936 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25930211811328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25069210077184 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13337395 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5979256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112677233374912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7358141 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 856045535232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113332087072768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114188132608000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3084619 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13347552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113332070033408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7358141 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3084902 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 861020346368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7845074774780521e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113332087072768 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 855704104960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113531633223579 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13337395 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1772656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113651712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚749 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114187774138368 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5979254 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚59 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5979254 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113531990824317 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 854740410045 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113332070033408 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13347552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 _runtime‚ 651.053862948 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 854399848667 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5979256 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272C=jÚ&ÈÔM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1772656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113651712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚749 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114187774138368 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5979254 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚59 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5979254 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113531990824317 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 854740410045 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚113332070033408 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13347552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 _runtime‚ 651.053862948 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 854399848667 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5979256 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7368296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132263936 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25930211811328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25069210077184 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13337395 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5979256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112677233374912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7358141 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 856045535232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113332087072768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114188132608000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3084619 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13347552 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113332070033408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7358141 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3084902 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561378 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 861020346368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7845074774780521e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113332087072768 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 855704104960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113531633223579 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13337395 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 150669312;Œ»«0É+ _wandbruntime‚651  _runtime‚651¾ô ÿÊù 0loss/SRqqt1/assignment_loss‚0.96340856895277 1loss/SRqqt2/assignment_loss‚0.840965407191608 /loss/FBt1/detection_loss‚0.7334292430723888 trainer/global_step‚749 _step‚60 /loss/FRt1/assignment_loss‚3.236359960036976 0loss/FBt1/assignment_loss‚0.1256149888266118 % _timestamp‚1.7845074774790945e+09 _runtime‚ 651.057997086 1loss/SRqqt2/detection_loss‚0.7492232428528137 epoch‚1 %loss/total_loss‚13.77989926418394 0loss/FRt2/assignment_loss‚1.7510508723533191 2loss/FBt2/assignment_loss‚0.033407776667173844 /loss/FRt1/detection_loss‚0.6293482875044323 .loss/FRt2/detection_loss‚1.307101828108628 1loss/SRqqt1/detection_loss‚1.5843550649596685 /loss/FBt2/detection_loss‚1.8256340236575515ëuËý 2loss/FBt2/assignment_loss‚0.033407776667173844 /loss/FRt1/detection_loss‚0.6293482875044323 .loss/FRt2/detection_loss‚1.307101828108628 1loss/SRqqt1/detection_loss‚1.5843550649596685 /loss/FBt2/detection_loss‚1.8256340236575515 0loss/SRqqt1/assignment_loss‚0.96340856895277 1loss/SRqqt2/assignment_loss‚0.840965407191608 /loss/FBt1/detection_loss‚0.7334292430723888 trainer/global_step‚749 _step‚60 /loss/FRt1/assignment_loss‚3.236359960036976 0loss/FBt1/assignment_loss‚0.1256149888266118 % _timestamp‚1.7845074774790945e+09 _runtime‚ 651.057997086 1loss/SRqqt2/detection_loss‚0.7492232428528137 epoch‚1 %loss/total_loss‚13.77989926418394 0loss/FRt2/assignment_loss‚1.7510508723533191<É•©þ7Ì‚(:¬ ÞØõÒáÇîË% disk.nvme1c1n1.out‚851.9873046875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚1552.15283203125" proc.memory.rssMB‚ 2902.8359375 cpu‚25.57128540840018 disk.md0.in‚0 disk.md0.out‚ 24.66015625 disk.dm-1.in‚0! disk.dm-1.out‚564.01123046875 network.sent‚ 1.016839e+06) disk./.usagePercent‚76.05949442853681 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚851.9873046875 disk.nvme0c0n1.in‚0$ proc.memory.availableMB‚5.984375* proc.memory.percent‚17.717504501342773 network.recv‚219091% disk./.usageGB‚1167.7033882141113% disk.nvme0n1.out‚1552.15283203125 disk.dm-0.in‚0! disk.dm-0.out‚1787.9814453125 disk.rbd0.in‚0$ memory_percent‚99.96347427368164 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0ýY°`žÍ‚(:“ ÞØõÒ·ŒÍ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.575' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.85833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬$ì/΂(:¤ íØõÒÙ¢­Ë disk.rbd0.in‚0' disk.nvme0c0n1.out‚1553.14404296875 disk.nvme0n1.in‚0% disk.nvme0n1.out‚1553.14404296875 disk.md0.out‚25.0625* proc.memory.percent‚17.717504501342773 proc.cpu.threads‚17 network.recv‚223358 disk.nvme0c0n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0! disk.dm-0.out‚1788.6220703125$ memory_percent‚99.96109008789062! proc.memory.availableMB‚6.375" proc.memory.rssMB‚ 2902.8359375 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚852.60498046875 disk.dm-0.in‚0 disk.dm-1.in‚0 cpu‚25.569415619018 network.sent‚ 1.079124e+06% disk./.usageGB‚1167.7034034729004& disk.nvme1c1n1.out‚852.60498046875 disk.dm-1.out‚564.0908203125) disk./.usagePercent‚76.05949542243286aW®‘Ï‚(:† íØõÒíÒ‰Ì gpu.0.gpu‚5.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.862' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.954 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðõ¹×:Ђ(:¯ üØõÒùàÚË" proc.memory.rssMB‚ 2902.8359375* proc.memory.percent‚17.717504501342773 network.sent‚ 1.085855e+06 disk.dm-0.out‚1797.244140625 disk.dm-1.in‚0% disk.nvme0n1.out‚1561.51708984375$ memory_percent‚99.96130466461182 cpu‚25.567559994812065 proc.cpu.threads‚17 network.recv‚226612$ disk.nvme1n1.out‚855.15283203125 disk.md0.out‚ 26.15234375 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0% disk./.usageGB‚1167.7028579711914 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.out‚564.1220703125 disk.rbd0.in‚0& disk.nvme1c1n1.out‚855.15283203125) disk./.usagePercent‚76.05945989064918 disk.rbd0.out‚0' disk.nvme0c0n1.out‚1561.51708984375 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0& proc.memory.availableMB‚ 6.33984375·ûƒÑ‚(:’ üØõÒϳœÌ gpu.0.gpu‚88.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.57' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.52333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hýó00Ò+ _wandbruntime‚692  _runtime‚692Ûf‰`ÄÓ¾ % _timestamp‚1.7845075190736291e+09 _runtime‚ 692.649160266 _step‚61 & lr-AdamW/pg1‚0.0006138613059824611 & lr-AdamW/pg2‚0.0006138613059824611 trainer/global_step‚799T¦8ÈÔ trainer/global_step‚799 % _timestamp‚1.7845075190736291e+09 _runtime‚ 692.649160266 _step‚61 & lr-AdamW/pg1‚0.0006138613059824611 & lr-AdamW/pg2‚0.0006138613059824611=Ëíõ0Õ+ _wandbruntime‚692  _runtime‚692ö.j3ÖÆO ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 910504530944 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7833137 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121507683746304 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6361111 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚121508042208768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6361111 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7833137 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3281551 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1396929 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1884907 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120597520638464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 915707136512 G?Device 4}7™ StatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6361113 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚799 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26695043311104 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 909115235089 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123882496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 910163107840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105270272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚120810468355489 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 _step‚62 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 910504530944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119901353120400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120597520638464 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6361113 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119901336081040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14204391 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7843278 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27610731835392 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184ÿ—ÐâÐ'×ÊO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151639 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14204391 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7843278 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27610731835392 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 910504530944 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7833137 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121507683746304 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6361111 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚121508042208768 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613312 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6361111 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7833137 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3281551 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516791 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1396929 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1884907 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120597520638464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 915707136512 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6361113 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚799 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26695043311104 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 909115235089 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123882496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 910163107840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105270272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚120810468355489 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561266 _step‚62 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 910504530944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119901353120400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120597520638464 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6361113 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119901336081040>Æ Î¿0Ø+ _wandbruntime‚693  _runtime‚6939|-¿Ö&ÙÐM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120745812179968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14211697 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚120959007411325 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27645022381568 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 916931064320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7852948 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1886967 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120048766748512 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1398663 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120745812179968 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6368904 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3285630 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 911614637056 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7842793 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14211697 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 909883062075 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1398666 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120958649810587 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚ 693.261331589 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121657426817024 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6368906 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1887262 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121657068347392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845075196855667e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26728204968960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14221854 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚63 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 911273206784 trainer/global_step‚799 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6368904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 911273206784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121657426817024 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 911614637056 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3285928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27645154645504 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 916817412608 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6368906 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7842793 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120745795140608 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120048783787872 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 910223623453 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176n ¥ÜnÚÔM TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121657426817024 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 911614637056 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3285928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27645154645504 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 916817412608 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6368906 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7842793 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120745795140608 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120048783787872 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 910223623453 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120745812179968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚14211697 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚120959007411325 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516791 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27645022381568 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 916931064320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358469632 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7852948 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1886967 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600738 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120048766748512 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613312 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613312 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1398663 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120745812179968 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6368904 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3285630 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 911614637056 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7842793 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14211697 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 909883062075 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1398666 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚120958649810587 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚ 693.261331589 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121657426817024 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6368906 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1887262 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121657068347392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845075196855667e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26728204968960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358469632 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14221854 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚63 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430272 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 911273206784 trainer/global_step‚799 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6368904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385472 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151639 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 911273206784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1?iE0Û+ _wandbruntime‚693  _runtime‚693ëÓ6×Üû /loss/FBt2/detection_loss‚1.2658604951791985 epoch‚1 $ _timestamp‚1.784507519686546e+09 _step‚64 0loss/FRt1/assignment_loss‚3.0114634374031573 2loss/SRqqt2/assignment_loss‚0.7841247825141275 1loss/FBt2/assignment_loss‚0.13138582230482804 /loss/FRt1/detection_loss‚0.5656420982839323 1loss/SRqqt1/detection_loss‚1.3577621271642146 1loss/SRqqt2/detection_loss‚0.7979624039511527 2loss/SRqqt1/assignment_loss‚0.7091994321839922 .loss/FRt2/detection_loss‚1.444449582057504 &loss/total_loss‚12.361990911695935 trainer/global_step‚799 _runtime‚ 693.26206725 0loss/FBt1/assignment_loss‚0.1609469715970282 /loss/FBt1/detection_loss‚0.6589839632933366 0loss/FRt2/assignment_loss‚1.4742097957634632…ÃŽ®Ýÿ trainer/global_step‚799 _runtime‚ 693.26206725 0loss/FBt1/assignment_loss‚0.1609469715970282 /loss/FBt1/detection_loss‚0.6589839632933366 0loss/FRt2/assignment_loss‚1.4742097957634632 /loss/FBt2/detection_loss‚1.2658604951791985 epoch‚1 $ _timestamp‚1.784507519686546e+09 _step‚64 0loss/FRt1/assignment_loss‚3.0114634374031573 2loss/SRqqt2/assignment_loss‚0.7841247825141275 1loss/FBt2/assignment_loss‚0.13138582230482804 /loss/FRt1/detection_loss‚0.5656420982839323 1loss/SRqqt1/detection_loss‚1.3577621271642146 1loss/SRqqt2/detection_loss‚0.7979624039511527 2loss/SRqqt1/assignment_loss‚0.7091994321839922 .loss/FRt2/detection_loss‚1.444449582057504 &loss/total_loss‚12.361990911695935@‡Jü4Þ‚(:© ‹ÙõÒ—ýÊË disk.nvme0c0n1.in‚0$ disk.nvme1n1.out‚856.27197265625 disk.dm-0.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.md0.in‚0 proc.memory.rssMB‚ 2909.4375 cpu‚25.608106973258643 network.sent‚ 1.10162e+06& disk.nvme0c0n1.out‚1751.0673828125$ disk.nvme0n1.out‚1751.0673828125 disk.md0.out‚ 26.74609375" disk.dm-0.out‚1986.28955078125 disk.dm-1.out‚ 564.486328125& proc.memory.availableMB‚ 1.96484375* proc.memory.percent‚17.757797241210938 network.recv‚230040% disk./.usageGB‚1169.7608604431152 disk.rbd0.in‚0 disk.nvme0n1.in‚0& disk.nvme1c1n1.out‚856.27197265625 disk.nvme1n1.in‚0 disk.dm-1.in‚0$ memory_percent‚99.98800754547119( disk./.usagePercent‚76.1935098806784Þ,Å"“ß‚(:ˆ ‹ÙõÒÉ™“Ì gpu.0.gpu‚50.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚288.159' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã]É«0à+ _wandbruntime‚708  _runtime‚708µ\ðR’=áŒ{ 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9940568804740906 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12664267420768738 jet/accuracy_4_of_5‚0 7"Purity/*FRt0SRqqt2FBt/event_purity‚0.85362708568573 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9936911463737488 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.12185981124639511 5 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.99150151014328 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5411580204963684 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11887049674987793 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_2_of_5‚1 $ _timestamp‚1.784507535398585e+09 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6988242268562317 particle/accuracy_5_of_5‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_3_of_3‚0.048072487115859985 *jet/accuracy_2_of_4‚0.7199467420578003 particle/accuracy_1_of_4‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.992004930973053 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32405927777290344 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731483578681946 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9847342371940613 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4887852370738983 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12020152062177658 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03265320509672165 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_5‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9961667656898499 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5540059208869934 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6523610353469849 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.12214431166648865 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21526215970516205 particle/accuracy_1_of_3‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23642566800117493 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.28346386551856995 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9237174987792969 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7975081205368042 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1290753334760666 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5225812792778015 particle/accuracy_2_of_2‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.192831352353096 'particle/f_score‚0.5975432395935059 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.866335928440094 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6184749603271484 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 _step‚65 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9814077019691467 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1323740929365158 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9925053119659424 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5995782017707825 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6546593904495239 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6593430638313293 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12420281767845154 *loss/val_total_loss‚12.164415394758741 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4346756339073181 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7008545398712158 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00684359110891819 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10040727257728577 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9868959784507751 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7748866081237793 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0074001336470246315 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13642320036888123 jet/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15327800810337067 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6003913879394531 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5728546977043152 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5194331407546997 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 trainer/global_step‚799 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5980852842330933 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12082064896821976 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.978264331817627 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003488868474960327 *jet/accuracy_1_of_2‚0.7348695993423462 jet/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08512020856142044 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.025532977655529976 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6377074122428894 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13353365659713745 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9961667656898499 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.014471855945885181 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.10864494740962982 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3248591125011444 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1314874142408371 +particle/specificity‚0.9040766954421997 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6845529675483704 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9910158514976501 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5310433506965637 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.409722238779068 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9259259700775146 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6410244703292847 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6574074029922485 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6788406372070312 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18204839527606964 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5540059208869934 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06376558542251587 /particle/accuracy_3_of_3‚0.9990025758743286 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 'particle/accuracy‚0.833523154258728 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7878686189651489 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7290431261062622 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5836658477783203 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9119638800621033 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6040089726448059 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5519835948944092 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1093851625919342 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15943066775798798 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_5_of_5‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4205802381038666 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3217412233352661 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1313176304101944 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8242954015731812 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9853717088699341 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9955812096595764 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.969468355178833 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5004637241363525 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11804427206516266 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19792760908603668 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12420281767845154 +jet/accuracy_1_of_1‚0.25909343361854553 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1506374627351761 *jet/accuracy_1_of_3‚0.8940978050231934 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1822916716337204 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9894105195999146 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3721844553947449 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971743226051331 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16733714938163757 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.291872501373291 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5266192555427551 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +jet/accuracy_2_of_2‚0.11663135141134262 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10494273900985718 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5222498774528503 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6333662867546082 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1093851625919342 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20147380232810974 particle/accuracy_1_of_2‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9867578148841858 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5 _runtime‚ 708.974689051 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.48111584782600403 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5489820241928101 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5455662608146667 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.47659918665885925 .particle/accuracy_3_of_4‚0.987941563129425 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5628307461738586 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6571654081344604 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5382275581359863 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4670833647251129 7validation_average_jet_accuracy‚0.30911514163017273 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43279820680618286 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_2_of_3‚1 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7576719522476196 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5514377355575562 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09441899508237839 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5983254313468933 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5362564921379089 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6574074029922485 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7290431261062622 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7660619616508484 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.979607880115509 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14555126428604126 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6802016496658325 *jet/accuracy_1_of_4‚0.9642302989959717 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5052083730697632 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6010697484016418 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5525907874107361 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13179242610931396 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1336745023727417 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08666666597127914 ,jet/accuracy_4_of_4‚0.007130124606192112 +particle/sensitivity‚0.5755386352539062 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1Ù“/û– â{ jet/accuracy_5_of_5‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4205802381038666 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3217412233352661 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1313176304101944 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8242954015731812 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9853717088699341 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9955812096595764 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.969468355178833 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5004637241363525 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11804427206516266 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19792760908603668 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12420281767845154 +jet/accuracy_1_of_1‚0.25909343361854553 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1506374627351761 *jet/accuracy_1_of_3‚0.8940978050231934 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1822916716337204 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09259259700775146 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9894105195999146 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3721844553947449 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09259259700775146 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971743226051331 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16733714938163757 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.291872501373291 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5266192555427551 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +jet/accuracy_2_of_2‚0.11663135141134262 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10494273900985718 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5222498774528503 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6333662867546082 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1093851625919342 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20147380232810974 particle/accuracy_1_of_2‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9867578148841858 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5 _runtime‚ 708.974689051 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.48111584782600403 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5489820241928101 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5455662608146667 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.47659918665885925 .particle/accuracy_3_of_4‚0.987941563129425 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5628307461738586 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6571654081344604 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5382275581359863 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4670833647251129 7validation_average_jet_accuracy‚0.30911514163017273 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43279820680618286 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_2_of_3‚1 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7576719522476196 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5514377355575562 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09441899508237839 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5983254313468933 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5362564921379089 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6574074029922485 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7290431261062622 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7660619616508484 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.979607880115509 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14555126428604126 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6802016496658325 *jet/accuracy_1_of_4‚0.9642302989959717 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5052083730697632 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6010697484016418 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5525907874107361 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13179242610931396 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1336745023727417 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08666666597127914 ,jet/accuracy_4_of_4‚0.007130124606192112 +particle/sensitivity‚0.5755386352539062 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9940568804740906 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12664267420768738 jet/accuracy_4_of_5‚0 7"Purity/*FRt0SRqqt2FBt/event_purity‚0.85362708568573 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9936911463737488 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.12185981124639511 5 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.99150151014328 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5411580204963684 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11887049674987793 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_2_of_5‚1 $ _timestamp‚1.784507535398585e+09 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6988242268562317 particle/accuracy_5_of_5‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_3_of_3‚0.048072487115859985 *jet/accuracy_2_of_4‚0.7199467420578003 particle/accuracy_1_of_4‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.992004930973053 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32405927777290344 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731483578681946 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9847342371940613 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4887852370738983 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12020152062177658 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03265320509672165 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 =&Purity/0FRt*S÷ËDRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_5‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9961667656898499 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5540059208869934 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6523610353469849 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.12214431166648865 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21526215970516205 particle/accuracy_1_of_3‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23642566800117493 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.28346386551856995 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9237174987792969 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7975081205368042 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1290753334760666 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5225812792778015 particle/accuracy_2_of_2‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.192831352353096 'particle/f_score‚0.5975432395935059 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.866335928440094 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6184749603271484 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 _step‚65 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9814077019691467 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1323740929365158 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9925053119659424 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5995782017707825 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6546593904495239 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6593430638313293 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12420281767845154 *loss/val_total_loss‚12.164415394758741 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4346756339073181 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7008545398712158 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00684359110891819 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10040727257728577 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9868959784507751 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7748866081237793 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0074001336470246315 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13642320036888123 jet/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15327800810337067 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6003913879394531 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5728546977043152 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5194331407546997 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 trainer/global_step‚799 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5980852842330933 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12082064896821976 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.978264331817627 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003488868474960327 *jet/accuracy_1_of_2‚0.7348695993423462 jet/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08512020856142044 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.025532977655529976 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6377074122428894 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13353365659713745 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9961667656898499 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.014471855945885181 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.10864494740962982 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3248591125011444 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1314874142408371 +particle/specificity‚0.9040766954421997 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6845529675483704 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9910158514976501 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5310433506965637 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.409722238779068 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9259259700775146 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6410244703292847 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6574074029922485 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6788406372070312 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18204839527606964 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5540059208869934 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06376558542251587 /particle/accuracy_3_of_3‚0.9990025758743286 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 'particle/accuracy‚0.833523154258728 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7878686189651489 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7290431261062622 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5836658477783203 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9119638800621033 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6040089726448059 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5519835948944092 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1093851625919342 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15943066775798798 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375A1 «>ã‚(:³ šÙõÒÊèÝË' disk.nvme0c0n1.out‚3482.40771484375 disk.nvme0n1.in‚0' proc.memory.availableMB‚ 492.5859375% disk./.usageGB‚1170.2256965637207 disk.md0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.798590660095215 cpu‚25.617439255264525 network.sent‚ 1.164026e+06 disk.dm-0.in‚0 disk.dm-0.out‚3717.423828125 disk.dm-1.out‚1055.275390625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ memory_percent‚96.99349403381348 network.recv‚234011) disk./.usagePercent‚76.22378743290804 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚3482.40771484375' disk.nvme1c1n1.out‚1348.01708984375% disk.nvme1n1.out‚1348.01708984375# proc.memory.rssMB‚ 2916.12109375 proc.cpu.threads‚17 disk.md0.out‚ 27.28515625 disk.dm-1.in‚0Ì“Å.œä‚(:‘ šÙõÒÜÈžÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚243.953' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚81.31766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼Fc4å‚(:© ©ÙõÒâ§Ë) disk./.usagePercent‚76.20834676086893 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 493.3359375 cpu‚25.614458804313877 proc.cpu.threads‚17 network.sent‚ 1.209795e+06% disk./.usageGB‚1169.9886436462402 disk.md0.out‚ 27.70703125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚3483.69677734375! disk.nvme1c1n1.out‚ 1348.65625# proc.memory.rssMB‚ 2915.12109375 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme1n1.out‚ 1348.65625 network.recv‚238259 disk.md0.in‚0 disk.dm-0.out‚3718.373046875" disk.dm-1.out‚1055.34814453125% disk.nvme0n1.out‚3483.69677734375$ memory_percent‚96.98891639709473* proc.memory.percent‚17.792487144470215]Ÿœæ‚(:‘ ©ÙõÒï¤óË gpu.0.gpu‚27.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚284.52' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.83999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý”I%Aç‚(:¶ ¸ÙõÒ—·ËË disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚1479.63134765625 network.recv‚241544 disk.dm-1.in‚0" disk.dm-1.out‚1055.37939453125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 27.93359375 disk.nvme0n1.in‚0$ memory_percent‚96.99358940124512* proc.memory.percent‚17.798638343811035 cpu‚25.61187224045422 network.sent‚ 1.212717e+06% disk./.usageGB‚1169.7039833068848' disk.nvme1c1n1.out‚1479.63134765625' proc.memory.availableMB‚ 492.5703125# proc.memory.rssMB‚ 2916.12890625 proc.cpu.threads‚17) disk./.usagePercent‚76.18980513316299 disk.dm-0.in‚0" disk.dm-0.out‚4616.25146484375 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚4251.15185546875% disk.nvme0n1.out‚4251.15185546875˜¥s“è‚(:ˆ ¸ÙõÒ³åÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚123.369' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚41.123 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á°n0é+ _wandbruntime‚750  _runtime‚750æWÂê¼ _runtime‚ 750.911578592 _step‚66 % lr-AdamW/pg1‚0.000613694092091399 % lr-AdamW/pg2‚0.000613694092091399 trainer/global_step‚849 % _timestamp‚1.7845075773359656e+09曉¤ÆëÀ trainer/global_step‚849 % _timestamp‚1.7845075773359656e+09 _runtime‚ 750.911578592 _step‚66 % lr-AdamW/pg1‚0.000613694092091399 % lr-AdamW/pg2‚0.000613694092091399Bqñy0ì+ _wandbruntime‚750  _runtime‚750fѦ—Ì'íÆO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚849 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129953589621248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129953589621248 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3530244 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8399283 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15282996 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129194142502224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845075773368683e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2042192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129953606660608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 986391469056 _runtime‚ 750.916014179 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129953606660608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6873569 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 986050045440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992377155584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1487748 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29369012633088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚129194125462864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130179044799619 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚130939998129664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8399283 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28376635477504 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 984902297395 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130939639666688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8409425 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚130939998129664 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29368890848256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1487745 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2042496 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 992273982976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130939639666688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28376616865280 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3529937 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 _step‚67 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 986050045440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 986391469056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GÐ'îÊO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3529937 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 _step‚67 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 986050045440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172608 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 986391469056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚849 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129953589621248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129953589621248 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3530244 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8399283 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15282996 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129194142502224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845075773368683e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2042192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129953606660608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 986391469056 _runtime‚ 750.916014179 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129953606660608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6873569 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 986050045440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992377155584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1487748 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29369012633088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚129194125462864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130179044799619 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚130939998129664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8399283 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28376635477504 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 984902297395 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130939639666688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8409425 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚130939998129664 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29368890848256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1487745 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2042496 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 992273982976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130939639666688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28376616865280C=Ÿb0ï+ _wandbruntime‚751  _runtime‚751ÓÙ…>eðÐM F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15290301 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 987501575168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3533945 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993493603328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103165440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 987160144384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15300459 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129341556130336 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6881364 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 993390437888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8408939 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130101864123392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130327583855455 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1489479 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 985670124377 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131089024267776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚129341573169696 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 986010685759 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6881364 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8419095 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15300459 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1489482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131089382737920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131089382737920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3534246 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344ÂNÆÓÚ&ñÔM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1489482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131089382737920 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131089382737920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3534246 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15290301 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 987501575168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3533945 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993493603328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103165440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 987160144384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15300459 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129341556130336 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6881364 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 993390437888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8408939 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130101864123392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130327583855455 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1489479 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 985670124377 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131089024267776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚129341573169696 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 986010685759 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6881364 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8419095 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777664 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15300459 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 _step‚71 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16147154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16157298 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30035630369280 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1040044949533 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8883935 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2157574 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2157860 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113396224 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7263221 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138408933875712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚138409292338688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1041960570880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8883935 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1048281978880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1041619147264 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31083930960384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 792.077988663 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137606061386627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚138409292338688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137367314728448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1041619147264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137367331767808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137367314728448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚899 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8894077 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31083817564160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137605703785997 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7263219 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.laS—Æ?rge_pool.allocated‚1574448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136565675875824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3732308 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1041960570880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30035648981504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7845076185024955e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360YHÖ#×'þÑO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8883935 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2157574 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2157860 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113396224 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7263221 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138408933875712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚138409292338688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1041960570880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8883935 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1048281978880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1041619147264 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31083930960384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 792.077988663 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137606061386627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚138409292338688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137367314728448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1041619147264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137367331767808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137367314728448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚899 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8894077 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31083817564160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137605703785997 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7263219 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1574448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136565675875824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3732308 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1041960570880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30035648981504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7845076185024955e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1048187194880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1040385510803 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚138408933875712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7263221 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 _step‚71 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16147154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16157298 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30035630369280 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1040044949533 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272Gñ1f0ÿ+ _wandbruntime‚792  _runtime‚792yÁóÝ&€×M QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7271014 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3736342 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7271012 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7271012 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8903747 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109194752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚138558676946944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3736065 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137515589230592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚138558318476800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1049308094976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137754600442463 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7271014 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137515606269952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1042729246208 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31118118734336 trainer/global_step‚899 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30068810639360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1049398677504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1041153337785 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8893591 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845076191136763e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚138558676946944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30068829251584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137754242841721 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16174761 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1576179 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2159886CkÎá&ÛM F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137515589230592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136713089503936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1041493899167 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31118118734336 trainer/global_step‚899 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30068810639360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1049398677504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1041153337785 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8893591 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845076191136763e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚138558676946944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30068829251584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137754242841721 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16174761 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1576179 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2159886 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7271014 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3736342 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7271012 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7271012 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8903747 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109194752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚138558676946944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3736065 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137515589230592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚138558318476800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1049308094976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137754600442463 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7271014 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137515606269952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1042729246208 B˜<ƒý /loss/FRt1/detection_loss‚0.5822254961811305 /loss/FBt1/detection_loss‚0.5392256450871289 &loss/total_loss‚12.148023233521364 1loss/FBt2/assignment_loss‚0.24346776183125143 1loss/SRqqt2/detection_loss‚0.7298065636390993 _step‚73 .loss/FRt2/detection_loss‚1.487688889556449 epoch‚2 trainer/global_step‚899 % _timestamp‚1.7845076191154995e+09 _runtime‚ 792.690804299 /loss/FRt1/assignment_loss‚2.791300584343795 0loss/FRt2/assignment_loss‚1.3092985234525945 2loss/SRqqt1/assignment_loss‚0.7488426914144003 1loss/FBt1/assignment_loss‚0.12345882637676761 1loss/SRqqt1/detection_loss‚1.1377539006879136 /loss/FBt2/detection_loss‚1.7500565451383592 2loss/SRqqt2/assignment_loss‚0.7048978058124746©.{ ‘„ _step‚73 .loss/FRt2/detection_loss‚1.487688889556449 epoch‚2 trainer/global_step‚899 % _timestamp‚1.7845076191154995e+09 _runtime‚ 792.690804299 /loss/FRt1/assignment_loss‚2.791300584343795 0loss/FRt2/assignment_loss‚1.3092985234525945 2loss/SRqqt1/assignment_loss‚0.7488426914144003 1loss/FBt1/assignment_loss‚0.12345882637676761 1loss/SRqqt1/detect/ÓÕÐvion_loss‚1.1377539006879136 /loss/FBt2/detection_loss‚1.7500565451383592 2loss/SRqqt2/assignment_loss‚0.7048978058124746 /loss/FRt1/detection_loss‚0.5822254961811305 /loss/FBt1/detection_loss‚0.5392256450871289 &loss/total_loss‚12.148023233521364 1loss/FBt2/assignment_loss‚0.24346776183125143 1loss/SRqqt2/detection_loss‚0.7298065636390993ITþË?…‚(:´ åÙõÒ„òœË' disk.nvme1c1n1.out‚1571.06005859375 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2916.12890625 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 network.sent‚ 1.284483e+06 network.recv‚251721 disk.dm-1.in‚0! disk.dm-1.out‚1058.9677734375 disk.rbd0.in‚0& disk.nvme0c0n1.out‚4461.7314453125 disk.md0.in‚0 disk.md0.out‚ 29.2265625" disk.dm-0.out‚4911.88916015625$ memory_percent‚96.99327945709229( proc.memory.availableMB‚ 492.62109375* proc.memory.percent‚17.798638343811035 cpu‚25.604457928432407$ disk.nvme0n1.out‚4461.7314453125% disk./.usageGB‚1170.2318534851074 disk.nvme0c0n1.in‚0% disk.nvme1n1.out‚1571.06005859375 disk.dm-0.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚76.22418846996302 disk.rbd0.out‚0iŒlßž†‚(:“ åÙõÒã§éË gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚270.792' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.26399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿lB?‡‚(:´ ôÙõÒåÅ¿Ë disk.md0.in‚0 disk.md0.out‚ 29.67578125 disk.dm-0.in‚0 disk.dm-0.out‚5384.060546875 disk.dm-1.in‚0$ memory_percent‚96.99187278747559' proc.memory.availableMB‚ 492.8515625 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚4914.95556640625% disk.nvme0n1.out‚4914.95556640625' disk.nvme1c1n1.out‚1591.09716796875! disk.dm-1.out‚1059.0927734375 proc.cpu.threads‚17) disk./.usagePercent‚76.04363333846979% disk./.usageGB‚1167.4598808288574 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2916.12890625* proc.memory.percent‚17.798638343811035 cpu‚25.601333413663973% disk.nvme1n1.out‚1591.09716796875 network.sent‚ 1.347661e+06 network.recv‚255551 disk.rbd0.out‚0 disk.nvme0n1.in‚0|æOwˆ‚(:’ ôÙõÒ‘Ö€Ì gpu.0.gpu‚67.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.97' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.65666666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˽¯\=‰‚(:² ƒÚõÒåøË disk.rbd0.out‚0 disk.nvme0n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚5384.662109375* proc.memory.percent‚17.798638343811035 network.sent‚ 1.350684e+06' disk.nvme0c0n1.out‚4915.99658203125 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚1591.9462890625' proc.memory.availableMB‚ 246.5703125% disk.nvme0n1.out‚4915.99658203125 network.recv‚259257) disk./.usagePercent‚76.04362265408729$ disk./.usageGB‚1167.459716796875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0 disk.md0.out‚ 30.17578125# proc.memory.rssMB‚ 2916.12890625" disk.dm-1.out‚1059.33056640625& disk.nvme1c1n1.out‚1591.9462890625$ memory_percent‚98.49505424499512 cpu‚25.598651702571015 proc.cpu.threads‚17 disk.rbd0.in‚0‡+ÈTœŠ‚(:‘ ƒÚõÒÇ‹îË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚136.453' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚45.48433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ðÎ~0‹+ _wandbruntime‚834  _runtime‚834 "R ÃŒ½ & lr-AdamW/pg1‚0.0006133006282999795 & lr-AdamW/pg2‚0.0006133006282999795 trainer/global_step‚949 $ _timestamp‚1.784507660685418e+09 _runtime‚ 834.261013734 _step‚74$e†cÇÁ & lr-AdamW/pg2‚0.0006133006282999795 trainer/global_step‚949 $ _timestamp‚1.784507660685418e+09 _runtime‚ 834.261013734 _step‚74 & lr-AdamW/pg1‚0.0006133006282999795JíÐ"i0Ž+ _wandbruntime‚834  _runtime‚834OT<£Ô'ÎO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145878228084736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1097529672704 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9368587 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1095868724211 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143937209249424 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9378729 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17031600 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17021456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7652869 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2273354 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚949 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚145878228084736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7652869 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1104170561024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚145878586547712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113658368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1097529672704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _step‚75 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9378729 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143937192210064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1095528162941 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1097188249088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144781039835648 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528(AŠñØ'ÒO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚145878586547712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7652871 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1104056902656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3934212 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145033077973635 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145032720373005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3934502 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31694643873280 _runtime‚ 834.265033161 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9378729 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143937192210064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1095528162941 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1097188249088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144781039835648 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145878228084736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1097529672704 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9368587 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1095868724211 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143937209249424 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9378729 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17031600 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17021456 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7652869 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2273354 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚949 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚145878228084736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7652869 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1104170561024 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚145878586547712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113658368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1097529672704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _step‚75 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1098298348032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145181617029471 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31727824143360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1096977112575 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1098639778816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144929314337792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128069120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7660664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32832997169152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9388399 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17049063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7660662 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17049063 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2275659 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9378243 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1096636551193 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9378243 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1662879 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚144929314337792 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1098298348032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32833125238272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146027971155968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1105282482688 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7660664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_byte„’ö s.small_pool.allocated‚ 1098639778816 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3938244 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1096636551193 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9378243 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1662879 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚144929314337792 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1098298348032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32833125238272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146027971155968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1105282482688 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7660664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1098639778816 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3938244 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1098298348032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145181617029471 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31727824143360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1096977112575 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1098639778816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144929314337792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128069120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7660664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32832997169152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9388399 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17049063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7660662 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17049063 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2275659 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272L˜8ßg0”+ _wandbruntime‚834  _runtime‚834à?f•þ _runtime‚ 834.874864952 _step‚77 /loss/FRt1/assignment_loss‚2.821410846864355 1loss/SRqqt1/detection_loss‚1.0759576689451933 /loss/FBt2/detection_loss‚1.4804838225245476 trainer/global_step‚949 % _timestamp‚1.7845076612994268e+09 0loss/FRt2/assignment_loss‚1.3430063064848439 1loss/SRqqt2/assignment_loss‚0.697083800722414 2loss/FBt2/assignment_loss‚0.030679208055642386 /loss/FRt2/detection_loss‚1.5332473386951098 &loss/total_loss‚11.774598276666108 epoch‚2 0loss/FBt1/detection_loss‚0.43549349337396487 2loss/SRqqt1/assignment_loss‚0.8421770751270299 0loss/FBt1/assignment_loss‚0.1655310494749578 /loss/FRt1/detection_loss‚0.5609718217618075 1loss/SRqqt2/detection_loss‚0.7885558446362412@IËÝ–‚ trainer/global_step‚949 % _timestamp‚1.7845076612994268e+09 0loss/FRt2/assignment_loss‚1.3430063064848439 1loss/SRqqt2/assignment_loss‚0.697083800722414 2loss/FBt2/assignment_loss‚0.030679208055642386 /loss/FRt2/detection_loss‚1.5332473386951098 &loss/total_loss‚11.774598276666108 epoch‚2 0loss/FBt1/detection_loss‚0.43549349337396487 2loss/SRqqt1/assignment_loss‚0.8421770751270299 0loss/FBt1/assignment_loss‚0.1655310494749578 /loss/FRt1/detection_loss‚0.5609718217618075 1loss/SRqqt2/detection_loss‚0.7885558446362412 _runtime‚ 834.874864952 _step‚77 /loss/FRt1/assignment_loss‚2.821410846864355 1loss/SRqqt1/detection_loss‚1.0759576689451933 /loss/FBt2/detection_loss‚1.4804838225245476M_Ê6—‚(:« ’ÚõÒÂóŽË disk.md0.in‚0& proc.memory.availableMB‚ 246.515625) disk./.usagePercent‚76.05910929381857 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 5071.9765625 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0$ memory_percent‚98.49538803100586* proc.memory.percent‚17.798638343811035 cpu‚25.596054932122772 disk.rbd0.in‚0 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚1683.7314453125 disk.md0.out‚ 30.6484375" disk.dm-1.out‚1059.48291015625# proc.memory.rssMB‚ 2916.12890625 proc.cpu.threads‚17 network.sent‚ 1.355455e+06 network.recv‚262491% disk./.usageGB‚1167.6974754333496 disk.nvme0c0n1.in‚0! disk.nvme0n1.out‚ 5071.9765625& disk.nvme1c1n1.out‚1683.7314453125 disk.dm-0.in‚0" disk.dm-0.out‚5631.27490234375 disk.nvme0n1.in‚0X=“˜‚(:ˆ ’ÚõÒ¡ÌùË gpu.0.gpu‚65.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚290.703' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.901 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%”©d<™‚(:± ¡ÚõÒ‚‘žË disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚1684.26318359375" disk.dm-0.out‚5632.06005859375$ memory_percent‚98.49550724029541* proc.memory.percent‚17.798638343811035& disk.nvme0c0n1.out‚5072.9873046875 disk.md0.in‚0 disk.dm-0.in‚0 network.recv‚267067 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 30.94140625 disk.dm-1.in‚0 disk.dm-1.out‚ 1059.6171875 cpu‚25.59295902225688 proc.cpu.threads‚17 network.sent‚ 1.418755e+06% disk./.usageGB‚1167.6976356506348 disk.rbd0.out‚0$ disk.nvme0n1.out‚5072.9873046875' disk.nvme1c1n1.out‚1684.26318359375( proc.memory.availableMB‚ 246.49609375# proc.memory.rssMB‚ 2916.12890625) disk./.usagePercent‚76.05911972972706 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0@ž+žš‚(:“ ¡ÚõÒÄÚñË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.466' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.48866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó°”¾;›‚(:° °ÚõÒœ´©Ë disk.md0.in‚0 disk.dm-0.in‚0 network.sent‚ 1.424316e+06 disk.nvme0n1.in‚0% disk.nvme0n1.out‚5081.70654296875& disk.nvme1c1n1.out‚1685.9248046875$ disk.nvme1n1.out‚1685.9248046875* proc.memory.percent‚17.798638343811035 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0" disk.dm-0.out‚5641.53955078125 disk.dm-1.in‚0 disk.dm-1.out‚ 1059.671875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0' disk.nvme0c0n1.out‚5081.70654296875 disk.md0.out‚ 31.33984375 network.recv‚270496) disk./.usagePercent‚76.05911426329881$ disk./.usageGB‚1167.697551727295$ memory_percent‚98.49560260772705 disk.rbd0.out‚0( proc.memory.availableMB‚ 246.48046875# proc.memory.rssMB‚ 2916.12890625 cpu‚25.590533927486227 proc.cpu.threads‚17¬Â]]’œ‚(:‡ °ÚõÒ„ÁíË gpu.0.gpu‚31.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.964' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×-·C0+ _wandbruntime‚876  _runtime‚876“öÄž¾ & lr-AdamW/pg1‚0.0006130744036710662 & lr-AdamW/pg2‚0.0006130744036710662 trainer/global_step‚999 % _timestamp‚1.7845077026909692e+09 _runtime‚ 876.268015185 _step‚78ˆ•wÈŸÂ & lr-AdamW/pg2‚0.0006130744036710662 trainer/global_step‚999 % _timestamp‚1.7845077026909692e+09 _runtime‚ 876.268015185 _step‚78 & lr-AdamW/pg1‚0.0006130744036710662Nò¡Ê0 + _wandbruntime‚876  _runtime‚876æÞÔ'¡ÎO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2387614 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33353657377280 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1152757350912 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17905902 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33353675989504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1151011376349 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚152194764942848 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153347522293760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1152757350912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1159909154816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8042519 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845077026915374e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4135459 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153347880756736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1151351937619 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚152194781982208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8042521 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9863381 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153347522293760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚79 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 876.268077415 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34513690414080 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34513566532096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1160014424576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9853239 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153347880756736 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9853239 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8042519 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚152194764942848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 trainer/global_step‚999 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151308742623024 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8042521 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1153098774528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1747845 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1747848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2387887 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152460094560643 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151308725583664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1153098774528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚152194781982208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153347522293760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚79 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 876.268077415 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34513690414080 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34513566532096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1160014424576 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9853239 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153347880756736 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9853239 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8042519 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚152194764942848 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 trainer/global_step‚999 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151308742623024 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8042521 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269760 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1153098774528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1747845 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1747848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2387887 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152460094560643 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151308725583664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1153098774528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚152194781982208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocc”·°ations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17905902 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33353675989504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1151011376349 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚152194764942848 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153347522293760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1152757350912 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1159909154816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8042519 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845077026915374e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4135459 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153347880756736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1151351937619 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚152194781982208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8042521O]¼”q0£+ _wandbruntime‚876  _runtime‚876µ¥6Þ&¤ØM FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17923365 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152343039444992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 _step‚80 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1161130673664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚999 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9862895 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9862895 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚152343056484352 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4139750 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8050314 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17913207 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1152460325983 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17913207 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153497265364992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153497265364992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚ 876.876721682 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151456156251136 % _timestamp‚1.7845077033009784e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33386856259584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152608276015737 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1154208880640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8050314 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152343039444992 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1749582 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34547986933248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971968 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9873051 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9873051 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4139467 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2389888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1153867449856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚152343056484352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153496906894848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2390168 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8050312 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34547860961280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1153867449856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1161023313920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152608633616479 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17923365 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8050312 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8050314 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17913207 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1152460325983 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17913207 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153497265364992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153497265364992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚ 876.876721682 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151456156251136 % _timestamp‚1.7845077033009784e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33386856259584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152608276015737 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1154208880640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8050314 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152343039444992 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1749582 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34547986933248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971968 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9873051 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9873051 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4139467 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2389888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1153867449856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚152343056484352 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153496906894848 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2390168 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8050312 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34547860961280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1153867449856 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1161023313920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152608633616479 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17923365 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8050312 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17923365 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152343039444992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 _step‚80 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1161130673664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚999 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Píœxf0¦+ _wandbruntime‚876  _runtime‚876ï4û~þ§ø 2loss/SRqqt1/assignment_loss‚0.7364565320442327 1loss/FBt1/assignment_loss‚0.08966577660185206 &loss/total_loss‚12.730518438402783 epoch‚2 $ _timestamp‚1.784507703305538e+09 1loss/SRqqt2/detection_loss‚0.7398315241745106 _step‚81 0loss/FRt1/assignment_loss‚2.7573717316075905 0loss/FRt2/assignment_loss‚1.2512365187748824 /loss/FRt1/detection_loss‚0.5413479119650497 /loss/FRt2/detection_loss‚1.5824312309854396 0loss/SRqqt1/detection_loss‚1.070235467767195 /loss/FBt1/detection_loss‚1.9915348849675216 ,loss/FBt2/detection_loss‚1.2792393413838 _runtime‚ 876.8809585 1loss/SRqqt2/assignment_loss‚0.688820861651916 2loss/FBt2/assignment_loss‚0.002346656478792397 trainer/global_step‚999³„Îߨü _runtime‚ 876.8809585 1loss/SRqqt2/assignment_loss‚0.688820861651916 2loss/FBt2/assignment_loss‚0.002346656478792397 trainer/global_step‚999 2loss/SRqqt1/assignment_loss‚0.7364565320442327 1loss/FBt1/assignment_loss‚0.08966577660185206 &loss/total_loss‚12.730518438402783 epoch‚2 $ _timestamp‚1.784507703305538e+09 1loss/SRqqt2/detection_loss‚0.7398315241745106 _step‚81 0loss/FRt1/assignment_loss‚2.7573717316075905 0loss/FRt2/assignment_loss‚1.2512365187748824 /loss/FRt1/detection_loss‚0.5413479119650497 /loss/FRt2/detection_loss‚1.5824312309854396 0loss/SRqqt1/detection_loss‚1.070235467767195 /loss/FBt1/detection_loss‚1.9915348849675216 ,loss/FBt2/detection_loss‚1.2792393413838Qp{k;7©‚(:¬ ¿ÚõÒ’ªìË cpu‚25.588163753124427 disk.dm-0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0% disk./.usageGB‚1167.6967506408691 disk.md0.in‚0 disk.md0.out‚ 31.9140625 disk.dm-1.in‚0# disk.nvme1n1.out‚1686.794921875& proc.memory.availableMB‚ 246.421875* proc.memory.percent‚17.798638343811035 proc.cpu.threads‚17 network.sent‚ 1.440123e+06 network.recv‚274009) disk./.usagePercent‚76.05906208375633" disk.dm-0.out‚5642.10986328125 disk.nvme0c0n1.in‚0# disk.nvme0n1.out‚5082.783203125 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚1686.794921875" disk.dm-1.out‚1059.84033203125 disk.rbd0.in‚0% disk.nvme0c0n1.out‚5082.783203125 disk.nvme1n1.in‚0# memory_percent‚98.4959602355957# proc.memory.rssMB‚ 2916.12890625ꈢ⟪‚(:” ¿ÚõÒðÞµÌ gpu.0.gpu‚92.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚178.855' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚59.618333333333325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q ˜cž«‚(:“ ÎÚõÒ‘û®Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚226.182' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.39399999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ|Pà:¬‚(:¯ ÎÚõÒ€€ÜÍ) disk./.usagePercent‚76.05905885359418% disk.nvme1n1.out‚1688.52001953125 disk.dm-1.in‚0% disk.nvme0c0n1.out‚5088.548828125 disk.nvme0n1.in‚0 network.sent‚ 1.491219e+06 disk.md0.out‚ 32.48046875 disk.dm-0.out‚5648.326171875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0% disk./.usageGB‚1167.6967010498047 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚1688.52001953125 disk.nvme1n1.in‚0# disk.nvme0n1.out‚5088.548828125% proc.memory.availableMB‚ 246.40625# proc.memory.rssMB‚ 2916.12890625* proc.memory.percent‚17.798638343811035 disk.md0.in‚0 disk.dm-0.in‚0" disk.dm-1.out‚1059.90673828125 disk.rbd0.out‚0$ memory_percent‚98.49605560302734 cpu‚25.585792619394944 proc.cpu.threads‚17 network.recv‚278013÷X‡0­‚(:¥ ÝÚõÒ˜ÃëË proc.cpu.threads‚17 network.sent‚ 1.49609e+06 network.recv‚281688 disk.md0.in‚0 disk.dm-0.in‚0# proc.memory.rssMB‚ 2915.12890625 cpu‚25.583176165552302 disk.nvme0c0n1.in‚0% disk.nvme0n1.out‚5089.32373046875! disk.nvme1c1n1.out‚ 1689.03125 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 1689.03125' proc.memory.availableMB‚ 247.1484375$ disk./.usageGB‚1167.696304321289" disk.dm-1.out‚1059.99072265625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚5648.845703125$ memory_percent‚98.49152565002441) disk./.usagePercent‚76.05903301229696 disk.dm-1.in‚0' disk.nvme0c0n1.out‚5089.32373046875 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.md0.out‚ 32.796875* proc.memory.percent‚17.792534828186035Íکד®‚(:ˆ ÝÚõÒßÕ³Ì gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚269.034' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I(Í0¯+ _wandbruntime‚918  _runtime‚918\ŸÃ°½ % _timestamp‚1.7845077449892817e+09 _runtime‚ 918.564844222 _step‚82 % lr-AdamW/pg1‚0.000612828536172478 % lr-AdamW/pg2‚0.000612828536172478 trainer/global_step‚1049ãPæRDZÁ % lr-AdamW/pg1‚0.000612828536172478 % lr-AdamW/pg2‚0.000612828536172478 trainer/global_step‚1049 % _timestamp‚1.7845077449892817e+09 _runtime‚ 918.564844222 _step‚82RFöl0²+ _wandbruntime‚918  _runtime‚918Æb:ÆW³ÓO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159608490050048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1208667876352 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8432169 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159886753547021 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158680258957264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10348033 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1208326452736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160816816502784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10337891 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35012689493504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1215846977536 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4336230 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1834548 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 918.565823564 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1834545 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36228408394752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160817174965760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8432169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2501401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160817174965760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 trainer/global_step‚1049 % _timestamp‚1.7845077449902608e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18770060 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚159887111147651 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10337891 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4335946 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8432171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1208667876352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1208326452736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35012670881280 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8432171 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36228536471040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109464064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1215737513472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1206494589757 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160816816502784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159608507089408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159608490050048 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159886753547021 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158680258957264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10348033 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1208326452736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160816816502784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10337891 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35012689493504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1215846977536 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4336230 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1834548 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 918.565823564 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1834545 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36228408394752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160817174965760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8432169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2501401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160817174965760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 trainer/global_step‚1049 % _timestamp‚1.7845077449902608e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18770060 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚159887111147651 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10337891 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4335946 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8432171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1208667876352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1208326452736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35012670881280 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8432171 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36228536471040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109464064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1215737513472 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1206494589757 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160816816502784 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159608507089408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159608490050048 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1209436551680 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4339938 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1209436551680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4340220 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8439962 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1216958387200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚1049 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2503938 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159756781591552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1209777982464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18797667 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159756764552192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1836279 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160966559574016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10347547 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35045869763584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159756764552192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18787509 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10357703 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10357703 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10347547 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1209777982464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35045851151360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1207943539391ýÍk©k ·áM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160035292602745 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160966201103872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1207602978009 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1836282 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2503659 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128069120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1209436551680 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4339938 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1209436551680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4340220 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8439962 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1216958387200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚1049 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2503938 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159756781591552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1209777982464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18797667 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚159756764552192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1836279 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160966559574016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10347547 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35045869763584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159756764552192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18787509 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10357703 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10357703 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10347547 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1209777982464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35045851151360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1207943539391 _step‚84 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158827706664096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚167022215157248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10822543 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168286110711808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8821819 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1262318364435 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1264236978176 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4537478 trainer/global_step‚1099 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168286469174784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10822543 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1921248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1271723518976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1271624540672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167313770134029 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10832685 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167022232196608 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37943426516480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166051792330864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _runtime‚ 960.565104414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚167022215157248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37943308925952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166051809370224 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8821821 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8821819 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚167022215157248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10822543 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341423616 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168286110711808 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8821819 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1262318364435 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1264236978176 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4537478 trainer/global_step‚1099 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168286469174784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10822543 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1921248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1271723518976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1271624540672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167313770134029 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10832685 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167022232196608 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37943426516480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166051792330864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _runtime‚ 960.565104414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚167022215157248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37943308925952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166051809370224 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8821821 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8821819 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168435853783040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4541224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167462309189753 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167170506698752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19671969 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103165440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10842355 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10842355 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8829612 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2618245 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8829612 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1265005653504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2618504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166199222998336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1265347084288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8829614 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚88 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10832199 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167462666790495 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1265005653504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1272738421248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 961.172898687 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8829614 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4541486 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36704883267584 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167170506698752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167170489659392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Fˆ9•ç&ÉáM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19661811 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168435853783040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4541224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167462309189753 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167170506698752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19671969 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103165440 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10842355 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10842355 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8829612 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2618245 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8829612 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1265005653504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2618504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166199222998336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1265347084288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8829614 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚88 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10832199 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167462666790495 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1265005653504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1272738421248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 961.172898687 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8829614 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4541486 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36704883267584 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167170506698752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167170489659392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37977603076608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1922979 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1319464656384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9211471 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174435940264448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174741144321667 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9211471 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174435957303808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174435957303808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174435940264448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38330716501504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2007945 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2730708 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173423342743824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20528808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11317337 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11317337 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845078293449893e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9211469 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600630 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2007948 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39658163901952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚1002.921137978 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1319806080000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚91 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚175755763383808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1149 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11307195 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4738948 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173423325704464 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚Õ“jég272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚175755404920832 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4738653 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1327466012672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1317461016573 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175755404920832 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1319806080000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1317801577843 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39658283589632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1327567088128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2007948 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39658163901952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚1002.921137978 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1319806080000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚91 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚175755763383808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151643 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561385984 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚1149 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873613824 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11307195 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4738948 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚173423325704464 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚175755404920832 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4738653 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1327466012672 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1317461016573 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175755404920832 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1319806080000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341423616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1317801577843 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39658283589632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1327567088128 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873613824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1319464656384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9211471 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174435940264448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174741144321667 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9211471 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174435957303808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174435957303808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174435940264448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38330716501504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2007945 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2730708 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173423342743824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20528808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11317337 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11317337 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845078293449893e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292[ð"j2×- _wandbruntime‚1003  _runtime‚1003‰aáXâ&ØÜM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38363896771584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115486208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2009679 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174584214766592 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20546271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9219262 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11327007 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1328679505408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1320574755328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 trainer/global_step‚1149 _step‚92 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1320916186112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39692460790784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11327007 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174584231805952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1318909966207 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚175904789521920 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20546271 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1328582631424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2733036 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9219264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11316851 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9219264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 $ _timestamp‚1.784507829955686e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚175905147992064 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4742989 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1318569404825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1320916186112 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2009682 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2733307 _runtime‚1003.531503376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39692576276992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4742715 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚175905147992064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174584231805952 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174584214766592 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0»ÞHFæ&ÙàM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38363896771584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115486208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2009679 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174584214766592 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20546271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9219262 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11327007 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1328679505408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1320574755328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 trainer/global_step‚1149 _step‚92 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1320916186112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39692460790784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11327007 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174584231805952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1318909966207 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚175904789521920 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20546271 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1328582631424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2733036 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9219264 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11316851 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9219264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516795 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 $ _timestamp‚1.784507829955686e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚175905147992064 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4742989 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1318569404825 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1320916186112 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2009682 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2733307 _runtime‚1003.531503376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561385984 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39692576276992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4742715 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚175905147992064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873613824 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174584231805952 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174584214766592 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1383309108224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183224699129856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9601121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1375375181824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180794859078064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9601119 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182167803308045 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181849682411008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1375375181824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2094645 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41373020501504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9601119 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2845934 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183224699129856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125979136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39989711393280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183225057592832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚1199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚181849682411008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1383309108224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183224699129856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9601121 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1375375181824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358462976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180794859078064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516795 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9601119 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182167803308045 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 167675904 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181849682411008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1375375181824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2094645 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41373020501504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9601119 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2845934 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358462976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183224699129856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561385984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125979136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39989711393280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183225057592832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561270 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚1199_Ÿ “72é- _wandbruntime‚1045  _runtime‚1045«xYzêÝM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182316699964511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1376143857152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11801503 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1384535720448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21420573 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4944255 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183374083730944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181997956913152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1376143857152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚96 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9608912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2096379 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚181997939873792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117845504 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4944549 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180942289745536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21420573 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21410415 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183374442201088 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9608914 _runtime‚1045.886506488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 $ _timestamp‚1.784507872310673e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181997956913152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1374052618233 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2847876 ;5DeviceStatsMonitor.on_train_batch_end/num_device_freÏmÞ:i e‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182316342363769 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41407309538304 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180942306784896 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9608914 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚181997939873792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1384417874944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1374393179615 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11811659 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183374442201088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2096382 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 231756472327Ø Bç&ëáM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40022910275584 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21420573 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4944255 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341430784 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183374083730944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181997956913152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1376143857152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚96 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9608912 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2096379 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚181997939873792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117845504 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561382 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4944549 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180942289745536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21420573 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21410415 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183374442201088 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9608914 _runtime‚1045.886506488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470144 $ _timestamp‚1.784507872310673e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181997956913152 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470144 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151643 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1374052618233 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2847876 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182316342363769 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873613824 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341430784 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41407309538304 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180942306784896 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9608914 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚181997939873792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1384417874944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1374393179615 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11811659 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183374442201088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 167675904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561385984 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2096382 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182316699964511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1376143857152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11801503 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600742 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1384535720448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12939691543579102 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6481027603149414 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4652777910232544 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960519671440125 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9927319288253784 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚10.66695905509692 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952288269996643 ,jet/accuracy_4_of_4‚0.013899499550461769 /particle/accuracy_3_of_4‚0.9968253374099731 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5271746516227722 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08309651166200638 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11071883887052536 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9887163043022156 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9156420826911926 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6903694868087769 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5283548831939697 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6627305150032043 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5390625 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9945050477981567 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3381689488887787 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6044831275939941 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9917063117027283 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.12649314105510712 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8471088409423828 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6256741285324097 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8715721368789673 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09054066985845566 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5241090059280396 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1042228415608406 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009951921179890633 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_3‚0.8805540800094604 +jet/accuracy_3_of_4‚0.09065638482570648 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4627203345298767 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11217060685157776 (particle/accuracy‚0.8403242230415344 epoch‚2 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6172235012054443 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4884601831436157 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0716666653752327 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018971331417560577 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09086409211158752 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18616469204425812 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.11868353188037872 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7282879948616028 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8513606190681458 *jet/accuracy_1_of_4‚0.9463034272193909 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10564342886209488 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10126737505197525 *jet/accuracy_1_of_2‚0.7277399897575378 /particle/accuracy_2_of_4‚0.9968253374099731 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5719192624092102 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6169734597206116 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5340485572814941 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08744961768388748 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9863213896751404 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5826495289802551 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5941060185432434 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10126737505197525 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9848119020462036 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6127457618713379 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6825509667396545 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6812316179275513 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6254152059555054 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5706270337104797 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6169734597206116 *jet/accuracy_1_of_1‚0.2673133909702301 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11208319664001465 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003155552549287677 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7136296033859253 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5838643908500671 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.24647076427936554 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6519026756286621 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_5_of_5‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_1_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.12708935141563416 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8415721654891968 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952305555343628 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2537515461444855 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12459725141525269 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 % _timestamp‚1.7845078881170352e+09 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5869669318199158 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.986090898513794 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5982702970504761 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005845717620104551 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920990467071533 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08374060690402985 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5196134448051453 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02405695989727974 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6856098771095276 *jet/accuracy_3_of_3‚0.0709519013762474 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5696936845779419 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09428741037845612 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.548682689666748 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8434377312660217 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7379002571105957 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_1‚1 'particle/f_score‚0.6201350688934326 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2434467375278473 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5814294815063477 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11113886535167694 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011715417727828026 7validation_average_jet_accuracy‚0.31225132942199707 5 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.99396812915802 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2028079777956009 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09423712641000748 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4263159930706024 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6024191975593567 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960519671440125 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5826495289802551 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10756796598434448 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/specificity‚0.9041275978088379 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1927083432674408(²(ƒHóÝ{ =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9931755065917969 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9900637865066528 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.11987058073282242 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6121405959129333 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8411186933517456 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923627972602844 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5762138366699219 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7095552682876587 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7907596826553345 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5662156343460083 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11327897757291794 9"Purity/0FRt*SRqqt1FBt/SR¥‰ì›:qqt_purity‚0.5665261149406433 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30116114020347595 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4567045569419861 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.13210907578468323 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 _step‚98 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8609421253204346 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6621828079223633 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4858333468437195 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.13660100102424622 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.921484649181366 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_1_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10490269958972931 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11000832915306091 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6413304209709167 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10049193352460861 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 particle/accuracy_3_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9726430177688599 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12939691543579102 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6481027603149414 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4652777910232544 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960519671440125 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9927319288253784 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚10.66695905509692 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952288269996643 ,jet/accuracy_4_of_4‚0.013899499550461769 /particle/accuracy_3_of_4‚0.9968253374099731 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5271746516227722 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08309651166200638 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11071883887052536 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9887163043022156 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9156420826911926 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6903694868087769 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5283548831939697 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6627305150032043 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5390625 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9945050477981567 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3381689488887787 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6044831275939941 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9917063117027283 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.12649314105510712 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8471088409423828 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6256741285324097 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8715721368789673 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09054066985845566 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5241090059280396 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1042228415608406 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009951921179890633 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_3‚0.8805540800094604 +jet/accuracy_3_of_4‚0.09065638482570648 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4627203345298767 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11217060685157776 (particle/accuracy‚0.8403242230415344 epoch‚2 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6172235012054443 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4884601831436157 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0716666653752327 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018971331417560577 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09086409211158752 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18616469204425812 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.11868353188037872 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7282879948616028 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8513606190681458 *jet/accuracy_1_of_4‚0.9463034272193909 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970890283584595 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10564342886209488 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10126737505197525 *jet/accuracy_1_of_2‚0.7277399897575378 /particle/accuracy_2_of_4‚0.9968253374099731 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5719192624092102 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6169734597206116 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5340485572814941 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08744961768388748 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9863213896751404 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5826495289802551 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5941060185432434 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10126737505197525 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 jet/accuracy_2_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9848119020462036 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6127457618713379 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6825509667396545 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6812316179275513 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6254152059555054 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5706270337104797 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6169734597206116 *jet/accuracy_1_of_1‚0.2673133909702301 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11208319664001465 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003155552549287677 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7136296033859253 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5838643908500671 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.24647076427936554 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6519026756286621 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_5_of_5‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_1_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.12708935141563416 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8415721654891968 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952305555343628 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2537515461444855 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12459725141525269 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 % _timestamp‚1.7845078881170352e+09 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5869669318199158 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.986090898513794 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5982702970504761 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005845717620104551 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920990467071533 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08374060690402985 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5196134448051453 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02405695989727974 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6856098771095276 *jet/accuracy_3_of_3‚0.0709519013762474 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5696936845779419 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09428741037845612 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.548682689666748 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8434377312660217 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7379002571105957 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_1_of_1‚1 'particle/f_score‚0.6201350688934326 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2434467375278473 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5814294815063477 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11113886535167694 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011715417727828026 7validation_average_jet_accuracy‚0.31225132942199707 5 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.99396812915802 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2028079777956009 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09423712641000748 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4263159930706024 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6024191975593567 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960519671440125 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5826495289802551 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10756796598434448 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/specificity‚0.9041275978088379 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1927083432674408 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.30000001192092896 trainer/global_step‚1199 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3356279134750366 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9403126239776611 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37962964177131653 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9966061115264893 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4014013707637787 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 /particle/accuracy_3_of_3‚0.9994933605194092 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09542310982942581 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11217060685157776 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.11107676476240158 )jet/accuracy_2_of_4‚0.534332275390625 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_2_of_3‚0.41078048944473267 +particle/sensitivity‚0.6070095896720886 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11461001634597778 particle/accuracy_2_of_5‚1 _runtime‚1061.693069695bª… 8ô‚(:­ óÛõҬؓÌ) disk./.usagePercent‚76.05894132538663' disk.nvme0c0n1.out‚7591.01513671875 disk.md0.out‚ 36.921875" disk.dm-0.out‚8457.79052734375# proc.memory.rssMB‚ 2939.70703125 cpu‚25.605842006093795 proc.cpu.threads‚17% disk.nvme0n1.out‚7591.01513671875 disk.nvme1c1n1.in‚0 network.sent‚ 1.783866e+06 network.recv‚317446 disk.dm-1.in‚0 disk.dm-1.out‚ 1306.11328125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0$ memory_percent‚96.99718952178955( proc.memory.availableMB‚ 491.98046875) proc.memory.percent‚17.94254779815674% disk.nvme1c1n1.out‚2251.236328125$ disk./.usageGB‚1167.694896697998# disk.nvme1n1.out‚2251.236328125Båo}žõ‚(:“ óÛõÒ¦ÜÕË gpu.0.gpu‚26.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚53.0 gpu.0.powerWatts‚119.134' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.71133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žÍÁ9ö‚(:® ‚ÜõÒƼÑË* proc.memory.percent‚17.943596839904785 disk.rbd0.in‚0 disk.nvme0n1.in‚0 disk.dm-0.in‚0" disk.dm-0.out‚8471.68408203125) disk./.usagePercent‚75.88194160852773 disk.dm-1.in‚0$ disk.nvme0c0n1.out‚ 7603.36328125" disk.nvme0n1.out‚ 7603.36328125 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 37.28515625 proc.cpu.threads‚17 disk.dm-1.out‚1306.216796875 disk.nvme0c0n1.in‚0' disk.nvme1c1n1.out‚2253.68505859375' proc.memory.availableMB‚ 491.5390625# proc.memory.rssMB‚ 2939.87890625 cpu‚25.603734831485376 network.sent‚ 1.829794e+06 network.recv‚321415% disk./.usageGB‚1164.9775085449219 disk.rbd0.out‚0% disk.nvme1n1.out‚2253.68505859375 disk.md0.in‚0# memory_percent‚96.9998836517334m¤Ýzž÷‚(:“ ‚ÜõÒ¬«ÔË gpu.0.gpu‚91.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚144.851' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚48.28366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Í[Ø>ø‚(:³ ‘ÜõÒÓÿÆË proc.cpu.threads‚17 network.sent‚ 1.834563e+06% disk.nvme0n1.out‚7604.19482421875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚7604.19482421875# proc.memory.rssMB‚ 2939.87890625) disk./.usagePercent‚75.88194160852773 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚2499.5986328125 disk.dm-0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚1551.662109375$ memory_percent‚96.99838161468506( proc.memory.availableMB‚ 491.78515625 network.recv‚324874% disk./.usageGB‚1164.9775085449219 disk.nvme0n1.in‚0$ disk.nvme1n1.out‚2499.5986328125 disk.md0.in‚0" disk.dm-0.out‚8472.23486328125 disk.nvme0c0n1.in‚0* proc.memory.percent‚17.943596839904785 disk.nvme1n1.in‚0 disk.md0.out‚ 37.6328125 disk.rbd0.in‚0 cpu‚25.602113556314684V’¬Éžù‚(:“ ‘ÜõÒôæ„Ì gpu.0.gpu‚90.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚257.153' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.71766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eª—2ú- _wandbruntime‚1104  _runtime‚1104¢Å+ÆûÀ % _timestamp‚1.7845079312755687e+09 _runtime‚1104.851142433 _step‚99 & lr-AdamW/pg1‚0.0006116489722160829 & lr-AdamW/pg2‚0.0006116489722160829 trainer/global_step‚1249s›”¤ÊüÄ _step‚99 & lr-AdamW/pg1‚0.0006116489722160829 & lr-AdamW/pg2‚0.0006116489722160829 trainer/global_step‚1249 % _timestamp‚1.7845079312755687e+09 _runtime‚1104.851142433cõ¤À2ý- _wandbruntime‚1104  _runtime‚1104÷R»yß'þÙO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚192657013515264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚1249 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10113579 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10113577 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192656655051776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192657013515264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚191205751393792 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12357996 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43131310226944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2185464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5186185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41671303559680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚192656655051776 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1451262121472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845079312765396e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12368139 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191205751393792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22481718 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191536379752174 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚191205734354432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1450920697344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1449071853560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2185461 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚1104.852102725 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3001012 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3000724 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43131186345472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344Ebæ=® ÿÝO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12357996 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1459988055040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚190087665499248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1448731292286 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1451262121472 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1459882785792 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191536737352808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10113579 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845079312765396e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12368139 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚191205751393792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22481718 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191536379752174 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚191205734354432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1450920697344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1449071853560 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.lW !…5arge_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2185461 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚1104.852102725 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3001012 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3000724 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43131186345472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚192657013515264 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚1249 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10113579 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10113577 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192656655051776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192657013515264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚191205751393792 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12357996 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43131310226944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2185464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5186185 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41671303559680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚192656655051776 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1451262121472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5190187 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22499181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191354008856576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22489022 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22499181 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12377809 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10121372 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43165494833664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1452372227584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1449839680538 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚101 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109194240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191684918807898 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1450180241924 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5190187 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22499181 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191354008856576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22489022 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22499181 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12377809 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10121372 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43165494833664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1452372227584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1449839680538 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚101 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109194240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191684918807898 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1450180241924 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344e—ÓCP2ƒ- _wandbruntime‚1105  _runtime‚1105>†„ 2loss/SRqqt1/assignment_loss‚0.5580023847494701 epoch‚3 trainer/global_step‚1249 0loss/FRt1/assignment_loss‚2.6580720079552185 /loss/FRt2/assignment_loss‚1.042106922206983 2loss/SRqqt2/assignment_loss‚0.7315068253781197 1loss/FBt1/assignment_loss‚0.04692480297381555 1loss/FBt2/assignment_loss‚0.02206343082324683 1loss/SRqqt1/detection_loss‚1.2310529166903599 % _timestamp‚1.7845079318977742e+09 1loss/SRqqt2/detection_loss‚0.6590531051750259 /loss/FBt2/detection_loss‚1.1932415848686582 &loss/total_loss‚11.052534212160552 _runtime‚1105.473029135 /loss/FRt1/detection_loss‚0.6041314292973804 /loss/FRt2/detection_loss‚1.4726545228359913 /loss/FBt1/detection_loss‚0.8337242792062812 _step‚102e2 Ð …… 1loss/FBt2/assignment_loss‚0.02206343082324683 1loss/SRqqt1/detection_loss‚1.2310529166903599 % _timestamp‚1.7845079318977742e+09 1loss/SRqqt2/detection_loss‚0.6590531051750259 /loss/FBt2/detection_loss‚1.1932415848686582 &loss/total_loss‚11.052534212160552 _runtime‚1105.473029135 /loss/FRt1/detection_loss‚0.6041314292973804 /loss/FRt2/detection_loss‚1.4726545228359913 /loss/FBt1/detection_loss‚0.8337242792062812 _step‚102 2loss/SRqqt1/assignment_loss‚0.5580023847494701 epoch‚3 trainer/global_step‚1249 0loss/FRt1/assignment_loss‚2.6580720079552185 /loss/FRt2/assignment_loss‚1.042106922206983 2loss/SRqqt2/assignment_loss‚0.7315068253781197 1loss/FBt1/assignment_loss‚0.04692480297381555fºbaÎ8†‚(:­  ÜõÒ¾¨àË disk.md0.in‚0( proc.memory.availableMB‚ 491.73828125 network.recv‚328316) disk./.usagePercent‚75.88194632953395# disk./.usageGB‚1164.97758102417 disk.rbd0.in‚0% disk.nvme0n1.out‚7611.43408203125 disk.md0.out‚37.96875 disk.dm-0.out‚8479.974609375$ memory_percent‚96.99866771697998' disk.nvme0c0n1.out‚7611.43408203125 disk.nvme1c1n1.in‚0* proc.memory.percent‚17.943596839904785 cpu‚25.603629791416083 proc.cpu.threads‚17 network.sent‚ 1.850374e+06 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚0# proc.memory.rssMB‚ 2939.87890625" disk.dm-1.out‚1551.77685546875 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0% disk.nvme1c1n1.out‚2500.923828125# disk.nvme1n1.out‚2500.923828125“Ò5“‡‚(:ˆ  ÜõÒ³åõË gpu.0.gpu‚79.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚123.849' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚41.283 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ÓXÈ?ˆ‚(:´ ¯ÜõÒÐÔË# proc.memory.rssMB‚ 2939.87890625 proc.cpu.threads‚17 network.recv‚332833% disk./.usageGB‚1164.9773635864258 disk.nvme1n1.in‚0 disk.md0.out‚ 38.5078125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚0& proc.memory.availableMB‚ 245.453125 cpu‚25.60687239616587 network.sent‚ 1.901778e+06 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚2501.66162109375 disk.md0.in‚0 disk.dm-0.in‚0" disk.dm-1.out‚1551.85107421875* proc.memory.percent‚17.943596839904785) disk./.usagePercent‚75.88193216651527% disk.nvme0n1.out‚7612.72412109375% disk.nvme1n1.out‚2501.66162109375 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.50187301635742" disk.dm-0.out‚8480.78076171875' disk.nvme0c0n1.out‚7612.72412109375%äËž‰‚(:“ ¯ÜõÒ¢Ë”Ì gpu.0.gpu‚86.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚263.153' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.71766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*°ýL@Š‚(:µ ¾ÜõÒë¾²Ë disk.dm-1.in‚0 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 2939.87890625% disk./.usageGB‚1166.9836730957031 disk.md0.in‚0 cpu‚25.607228013364374' disk.nvme0c0n1.out‚7635.73583984375 disk.nvme0n1.in‚0! disk.dm-0.out‚8503.2841796875! disk.dm-1.out‚1551.9423828125 disk.nvme0c0n1.in‚0' disk.nvme1c1n1.out‚2503.05029296875 disk.nvme1n1.in‚0 disk.md0.out‚ 39.37890625 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1n1.out‚2503.05029296875$ memory_percent‚98.50192070007324' proc.memory.availableMB‚ 245.4453125* proc.memory.percent‚17.943596839904785 proc.cpu.threads‚17% disk.nvme0n1.out‚7635.73583984375 network.sent‚ 1.904742e+06 network.recv‚336117) disk./.usagePercent‚76.01261508521108 disk.dm-0.in‚0 €Ãòž‹‚(:“ ¾ÜõÒºéÿË gpu.0.gpu‚83.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚166.888' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚55.62933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÄ»U2Œ- _wandbruntime‚1148  _runtime‚1148fs×&ÇÁ % _timestamp‚1.7845079753680205e+09 _runtime‚1148.945022092 _step‚103 & lr-AdamW/pg1‚0.0006113051524385678 & lr-AdamW/pg2‚0.0006113051524385678 trainer/global_step‚1299lx’ŽÅ & lr-AdamW/pg2‚0.0006113051524385678 trainer/global_step‚1299 % _timestamp‚1.7845079753680205e+09 _runtime‚1148.945022092 `üH×9_step‚103 & lr-AdamW/pg1‚0.0006113051524385678g©³"2- _wandbruntime‚1148  _runtime‚1148[|›Þ'ØO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1504555066968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198619459461632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10503227 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚200126307724288 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚200125949260800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12842648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23356020 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198963396339182 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197459198872848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1506489799168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10503229 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43330317063680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12852791 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5388069 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198619476500992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1506831223296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚1148.945112413 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚200125949260800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚200126307724288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1506489799168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5388366 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1515896084992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 $ _timestamp‚1.784507975368748e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ãt¿â'‘ÜO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1504214505694 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10503229 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12852791 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚200125949260800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚200126307724288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1506489799168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5388366 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1515896084992 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 $ _timestamp‚1.784507975368748e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1504555066968 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198619459461632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10503227 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚200126307724288 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚200125949260800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12842648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23356020 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198963396339182 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197459198872848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1506489799168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10503229 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43330317063680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12852791 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5388069 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198619476500992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1506831223296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚1148.945112413 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10511022 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1507599898112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1505322893946 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3118451 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23373483 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23363324 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10511020 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198767751003136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1507941329408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198767733963776 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12862461 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197606612500960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2273898 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2273895 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43363497333760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1507941329408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚200275692332544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10511022 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5392349 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43363515945984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23373483 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚105 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3118180 _runtime‚ 1149.56492802 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1516912681472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199112292995652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199111935394906 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 % _timestamp‚1.7845079759892173e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚200275692332544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1517015846400 trainer/global_step‚1299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12852304 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198767733963776 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚44880410015232 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5392075 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5392075 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10511022 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1507599898112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1505322893946 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3118451 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23373483 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23363324 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10511020 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198767751003136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1507941329408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198767733963776 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12862461 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197606612500960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2273898 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2273895 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43363497333760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1507941329408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚200275692332544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10511022 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5392349 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43363515945984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23373483 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚105i}QÛñ2•- _wandbruntime‚1149  _runtime‚1149Š€×– 0loss/FRt2/assignment_loss‚1.0666781161648344 2loss/FBt1/assignment_loss‚0.023680399043518104 epoch‚3 /loss/FRt1/assignment_loss‚2.503751146547573 1loss/SRqqt2/assignment_loss‚0.668775014111385 2loss/FBt2/assignment_loss‚0.032309691954191866 1loss/SRqqt1/detection_loss‚1.1094341259383902 /loss/FBt1/detection_loss‚0.7866388960366776 &loss/total_loss‚10.636322699978514 _runtime‚1149.565533201 _step‚106 2loss/SRqqt1/assignment_loss‚0.6526692811031118 /loss/FRt1/detection_loss‚0.5609378536916206 .loss/FRt2/detection_loss‚1.570663144589361 1loss/SRqqt2/detection_loss‚0.6975278421219253 /loss/FBt2/detection_loss‚0.9632571886759251 trainer/global_step‚1299 % _timestamp‚1.7845079759902506e+09qVä —… _step‚106 2loss/SRqqt1/assignment_loss‚0.6526692811031118 /loss/FRt1/detection_loss‚0.5609378536916206 .loss/FRt2/detection_loss‚1.570663144589361 1loss/SRqqt2/detection_loss‚0.6975278421219253 /loss/FBt2/detection_loss‚0.9632571886759251 trainer/global_step‚1299 % _timestamp‚1.7845079759902506e+09 0loss/FRt2/assignment_loss‚1.0666781161648344 2loss/FBt1/assignment_loss‚0.023680399043518104 epoch‚3 /loss/FRt1/assignment_loss‚2.503751146547573 1loss/SRqqt2/assignment_loss‚0.668775014111385 2loss/FBt2/assignment_loss‚0.032309691954191866 1loss/SRqqt1/detection_loss‚1.1094341259383902 /loss/FBt1/detection_loss‚0.7866388960366776 &loss/total_loss‚10.636322699978514 _runtime‚1149.565533201jy»a>˜‚(:³ ÍÜõÒÚ©ÔË disk.md0.out‚ 39.734375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0& disk.nvme0c0n1.out‚9530.0537109375 network.sent‚ 1.923422e+06( disk./.usagePercent‚76.0618174120728' disk.nvme1c1n1.out‚2503.63427734375 disk.nvme1n1.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚1552.0478515625 disk.rbd0.out‚0 disk.nvme0n1.in‚0$ memory_percent‚98.50246906280518 cpu‚25.605415719313836% disk./.usageGB‚1167.7390518188477 disk.dm-0.in‚0# disk.dm-0.out‚10397.30322265625$ disk.nvme0n1.out‚9530.0537109375( proc.memory.availableMB‚ 245.35546875# proc.memory.rssMB‚ 2939.87890625* proc.memory.percent‚17.943596839904785 proc.cpu.threads‚17 network.recv‚339639 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚2503.63427734375 disk.md0.in‚0äQjœ™‚(:‘ ÍÜõÒ¨ÕšÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.453' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.81766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËÐI˜:š‚(:¯ ÜÜõÒ†§ÈË disk.dm-0.in‚0' disk.nvme1c1n1.out‚2504.18017578125 disk.nvme1n1.in‚0% proc.memory.availableMB‚ 245.34375# proc.memory.rssMB‚ 2939.87890625* proc.memory.percent‚17.943596839904785% disk.nvme0n1.out‚9532.81005859375 network.recv‚343426) disk./.usagePercent‚76.06182238155303$ disk./.usageGB‚1167.739128112793 disk.dm-1.in‚0! disk.dm-1.out‚1552.1689453125 disk.rbd0.in‚0% disk.nvme1n1.out‚2504.18017578125' disk.nvme0c0n1.out‚9532.81005859375 disk.nvme0n1.in‚0 disk.nvme1c1n1.in‚0 disk.md0.in‚0# memory_percent‚98.5025405883789 cpu‚25.604508494652833 disk.nvme0c0n1.in‚0! disk.dm-0.out‚10399.986328125 disk.rbd0.out‚0 disk.md0.out‚ 39.953125 proc.cpu.threads‚17 network.sent‚ 1.973679e+06§ñŸ¿ž›‚(:“ ÜÜõÒÈ—”Ì gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.336' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.44533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì,Æÿ@œ‚(:µ ëÜõÒ¦ÍÁË& proc.memory.availableMB‚ 245.171875% disk./.usageGB‚1167.6929321289062 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚11267.142578125# proc.memory.rssMB‚ 2939.87890625 cpu‚25.60299773404941% disk.nvme1n1.out‚2833.79541015625 disk.md0.in‚0 disk.dm-1.in‚0$ memory_percent‚98.50358963012695* proc.memory.percent‚17.943596839904785 network.recv‚346849) disk./.usagePercent‚76.05881336127057( disk.nvme0c0n1.out‚10183.11279296875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚2833.79541015625 disk.md0.out‚ 40.28515625! disk.dm-1.out‚1664.2158203125 proc.cpu.threads‚17 network.sent‚ 1.976601e+06 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0& disk.nvme0n1.out‚10183.11279296875 disk.nvme1n1.in‚0 disk.dm-0.in‚0ýM¬Ÿ‚(:” ëÜõÒÍ÷ÔË gpu.0.gpu‚38.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.481' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.160333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰åÚ”2ž- _wandbruntime‚1191  _runtime‚1191^G|ñÇŸÁ _runtime‚1191.936773177 _step‚107 & lr-AdamW/pg1‚0.0006109418034270783 & lr-AdamW/pg2‚0.0006109418034270783 trainer/global_step‚1349 % _timestamp‚1.7845080183612413e+09ʺWOË Å trainer/global_step‚1349 % _timestamp‚1.7845080183612413e+09 _runtime‚1191.936773177 _step‚107 & lr-AdamW/pg1‚0.0006109418034270783 & lr-AdamW/pg2‚0.0006109418034270783kª+æ]2¡- _wandbruntime‚1191  _runtime‚1191©@TÏß'¢ÙO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13327300 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206033184568832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚44989330567680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207595601933312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24230322 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3230821 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1559697719102 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46561112351232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10892879 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13327300 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3230530 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13337443 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207595243469824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚204830715207088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1571655804928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚1349 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1571763171328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13337443 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2358864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206033201608192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1562058900992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1562058900992 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207595243469824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0F×&€„"£ÝO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10892877 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1562400325120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206390770526824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10892879 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206390412926190 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2358861 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚44989349179904 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1560038280376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207595243469824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13327300 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206033184568832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚44989330567680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207595601933312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24230322 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3230821 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1559697719102 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46561112351232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10892879 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13327300 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3230530 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13337443 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207595243469824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚204830715207088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1571655804928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚1349 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1571763171328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13337443 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2358864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206033201608192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1562058900992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.al¨]J±_l.peak‚ 23873614336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1562058900992 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5593335 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13336956 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206181476110336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13336956 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45022529449984 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46595289751552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚204978162913920 trainer/global_step‚1349 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24237626 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784508018985476e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2360598 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3233021 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3232740 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10900670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206181476110336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24247785 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206538951981914 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1561146668740 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1563168999936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207744986541568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207744986541568 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1563510431232 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45022510837760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206181459070976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207744628070912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206181459070976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1563510431232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1572778913792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0=8H-ì&¦æM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1563168999936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207744986541568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207744986541568 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1563510431232 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45022510837760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206181459070976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207744628070912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚206181459070976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1563510431232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1572778913792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5593335 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13336956 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206181476110336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13336956 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45022529449984 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46595289751552 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚204978162913920 trainer/global_step‚1349 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24237626 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784508018985476e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2360598 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3233021 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3232740 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10900670 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206181476110336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24247785 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚206538951981914 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1561146668740 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528m2Ùtþ2§- _wandbruntime‚1192  _runtime‚1192‰EXŽ¨ .loss/FRt2/detection_loss‚1.602584904508929 1loss/SRqqt1/detection_loss‚1.0550401861417211 1loss/SRqqt2/detection_loss‚0.6978924378975501 /loss/FBt1/detection_loss‚0.5150784726836517 epoch‚3 $ _timestamp‚1.784508018987441e+09 /loss/FRt1/assignment_loss‚2.383696248501907 &loss/total_loss‚10.352176900515513 trainer/global_step‚1349 2loss/SRqqt2/assignment_loss‚0.6814796107630757 2loss/FBt1/assignment_loss‚0.029364502498412135 /loss/FRt1/detection_loss‚0.5983649254915918 /loss/FBt2/detection_loss‚1.0663253236562014 0loss/FRt2/assignment_loss‚1.0723551666642737 _runtime‚1192.562937384 _step‚110 2loss/SRqqt1/assignment_loss‚0.6403742314054802 2loss/FBt2/assignment_loss‚0.009620890302720144ãÚ ©… 0loss/FRt2/assignment_loss‚1.0723551666642737 _runtime‚1192.562937384 _step‚110 2loss/SRqqt1/assignment_loss‚0.6403742314054802 2loss/FBt2/assignment_loss‚0.009620890302720144 .loss/FRt2/detection_loss‚1.602584904508929 1loss/SRqqt1/detection_loss‚1.0550401861417211 1loss/SRqqt2/detection_loss‚0.6978924378975501 /loss/FBt1/detection_loss‚0.5150784726836517 epoch‚3 $ _timestamp‚1.784508018987441e+09 /loss/FRt1/assignment_loss‚2.383696248501907 &loss/total_loss‚10.352176900515513 trainer/global_step‚1349 2loss/SRqqt2/assignment_loss‚0.6814796107630757 2loss/FBt1/assignment_loss‚0.029364502498412135 /loss/FRt1/detection_loss‚0.5983649254915918 /loss/FBt2/detection_loss‚1.0663253236562014n-†.£<ª‚(:± úÜõÒÞéÒË disk.nvme1c1n1.in‚0# disk.dm-0.out‚11272.25146484375 disk.nvme0c0n1.in‚0# memory_percent‚98.5025405883789* proc.memory.percent‚17.943596839904785% disk.nvme0n1.out‚10187.9326171875 disk.nvme1n1.in‚0% disk.nvme1n1.out‚3258.06689453125 disk.md0.in‚0 disk.md0.out‚ 40.8203125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 1.99427e+06 disk.nvme0n1.in‚0 disk.dm-0.in‚0% proc.memory.availableMB‚ 245.34375# proc.memory.rssMB‚ 2939.87890625 cpu‚25.60172141985638 network.recv‚350061' disk.nvme1c1n1.out‚3258.06689453125! disk.dm-1.out‚2087.0654296875' disk.nvme0c0n1.out‚10187.9326171875 proc.cpu.threads‚17) disk./.usagePercent‚76.05880590705021% disk./.usageGB‚1167.6928176879883a¤@Éž«‚(:“ úÜõÒ„ÏÌ gpu.0.gpu‚93.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.546' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.84866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï^8¬‚(:­ ‰ÝõÒ¼ðšË disk.md0.out‚ 41.13671875# disk.dm-0.out‚11272.79833984375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚3258.53271484375* proc.memory.percent‚17.943596839904785 proc.cpu.threads‚17! disk.dm-1.out‚2087.1044921875# disk.nvme0c0n1.out‚ 10188.734375# memory_percent‚98.5027551651001( proc.memory.availableMB‚ 245.30859375# proc.memory.rssMB‚ 2939.87890625 disk.rbd0.out‚0 network.sent‚ 2.043779e+06% disk./.usageGB‚1166.2903327941895 disk.nvme1n1.in‚0% disk.nvme1n1.out‚3258.53271484375 disk.dm-0.in‚0 cpu‚25.59989216357467 network.recv‚354383( disk./.usagePercent‚75.9674536912027 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0! disk.nvme0n1.out‚ 10188.734375 disk.md0.in‚0²ò“­‚(:ˆ ‰ÝõÒ¡ÚóË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚135.684' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚45.228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£§H@®‚(:µ ˜ÝõÒ§ØÑË network.sent‚ 2.050539e+06 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚3259.7080078125 disk.nvme0c0n1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚1166.290428161621 disk.nvme0n1.in‚0$ disk.nvme1n1.out‚3259.7080078125 disk.md0.in‚0" disk.dm-1.out‚2087.25244140625 disk.rbd0.out‚0( proc.memory.availableMB‚ 245.16796875# proc.memory.rssMB‚ 2939.87890625 network.recv‚357720 disk.nvme1n1.in‚0" disk.dm-0.out‚11276.5732421875 disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚10192.03662109375$ memory_percent‚98.50361347198486* proc.memory.percent‚17.943596839904785' disk./.usagePercent‚75.967459903053& disk.nvme0n1.out‚10192.03662109375 disk.md0.out‚ 41.38671875 disk.dm-0.in‚0 cpu‚25.598270843757707)¼"|ž¯‚(:“ ˜ÝõÒ©÷Ì gpu.0.gpu‚49.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.205' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.40166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f2°- _wandbruntime‚1235  _runtime‚1235¬óŠŸÇ±Á & lr-AdamW/pg1‚0.0006105589485188425 & lr-AdamW/pg2‚0.0006105589485188425 trainer/global_step‚1399 % _timestamp‚1.7845080615732605e+09 _runtime‚1235.150019898 _step‚111çÃàŽË²Å & lr-AdamW/pg1‚0.0006105589485188425 & lr-AdamW/pg2‚0.0006105589485188425 trainer/global_step‚1399 % _timestamp‚1.7845080615732605e+09 _runtime‚1235.150019898 _step‚111ohsÿI2³- _wandbruntime‚1235  _runtime‚1235wÐPá`´ÙO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2445564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1615180932510 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1617969426944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1617628002816 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13822095 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1617628002816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3345155 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212202265620048 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11282527 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215064537678848 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.alloca2ª¸ted‚25104624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11282527 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215064896142336 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1615521493784 _step‚112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2445561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11282529 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215064896142336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213446926715392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11282529 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213817429513198 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48275978593280 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213446909676032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1627512737280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13822095 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48275856808960 _runtime‚1235.150081248 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25094479 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46648344071680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1627615909376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25104624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11282527 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215064896142336 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1615521493784 _step‚112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2445561 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11282529 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215064896142336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213446926715392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11282529 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213817429513198 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48275978593280 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213446909676032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1627512737280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13822095 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48275856808960 _runtime‚1235.150081248 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25094479 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46648344071680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1627615909376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212202265620048 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11282527 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215064537678848 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13831765 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5794735 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13831765 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213595201217536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25111928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212349679248160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25122087 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215214280750592 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5795022 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212349696287520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1616629882148 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13821608 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25122087 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 _step‚113 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11290322 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1616289320762 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1628632821248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46681542953984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11290322 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11290320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13821608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1628733889024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213965968568922 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213595184178176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1618738101760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0,ì`¸çM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1619079533056 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215213922279936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2447298 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46681524341760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215214280750592 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _runtime‚1235.771355639 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13831765 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5794735 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13831765 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213595201217536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25111928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212349679248160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25122087 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚215214280750592 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5795022 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212349696287520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1616629882148 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13821608 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25122087 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 _step‚113 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11290322 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1616289320762 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1628632821248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46681542953984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11290322 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11290320 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13821608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1628733889024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚213965968568922 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213595184178176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1618738101760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25111928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11290320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2447295 [HDevMó|WiceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48310276843008 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48310157163008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222533831887872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3460238 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48307357575680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚222533831887872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1673197104640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25968781 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3459951 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14296604 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222534190351360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221244803700840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _step‚116 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1673538528768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11672177 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222534190351360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11672179 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚220860651822592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1671004707192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚220860634783232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11672177 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14296604 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221244446100206 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚1278.239088926 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5992212 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25978926 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2532261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1670664145918 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚220860651822592 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5992502 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49990805686784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚220860634783232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14306747 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1673538528768 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222533831887872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3460238 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48307357575680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚222533831887872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1673197104640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25968781 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3459951 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14296604 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222534190351360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚221244803700840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _step‚116 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1673538528768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11672177 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222534190351360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11672179 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚220860651822592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1671004707192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚220860634783232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11672177 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14296604 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221244446100206 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚1278.239088926 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5992212 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25978926 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2532261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1670664145918 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚220860651822592 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5992502 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128t\Ë)a2È- _wandbruntime‚1278  _runtime‚1278¢pðÉâM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221008909285376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50025102740480 % _timestamp‚1.7845081052843258e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚222683574959616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221393342756676 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221392985155930 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1674307203584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1674307203584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219721212621760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1672113095556 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221008909285376 É¢ïêÙ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48340537845760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1671772534170 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚1449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3462212 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50024966283264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11679972 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48340556457984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1684428437504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221008926324736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3462502 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11679972 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5996500 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1684546282496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11679970 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5996207 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222683216488960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25986230 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1674648634880 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25996389 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221008909285376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50025102740480 % _timestamp‚1.7845081052843258e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚222683574959616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221393342756676 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221392985155930 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1674307203584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1674307203584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219721212621760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1672113095556 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221008909285376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48340537845760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1671772534170 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚1449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3462212 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚50024966283264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11679972 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48340556457984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1684428437504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221008926324736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3462502 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11679972 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5996500 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1684546282496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11679970 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5996207 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222683216488960 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25986230 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1674648634880 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25996389 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230003126096896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49966389691904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1739135500800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚ 1321.22749686 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14791399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12061829 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12061827 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚230003484560384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226945315327888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14781256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1728766206464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2618964 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6193071 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14781256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49966371079680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 # _timestamp‚1.78450814765205e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3574110 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26853228 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚228274376929792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1739242867200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚228671462687214 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚228274359890432 trainer/global_step‚1499 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452#Š¦ÊÙÚO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YEDeviceStatsMJ‚$%onitor.on_train_batch_start/allocated_bytes.all.allocated‚230003484560384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228671820287848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230003126096896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49966389691904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1739135500800 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚ 1321.22749686 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14791399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12061829 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12061827 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚230003484560384 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226945315327888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14781256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1728766206464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2618964 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6193071 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14781256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49966371079680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 # _timestamp‚1.78450814765205e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3574110 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26853228 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚228274376929792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1739242867200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚228671462687214 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚228274359890432 trainer/global_step‚1499 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1729876305408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14790912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12069622 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚228422651431936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12069622 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14801069 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6197099 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1727255747578 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚1499 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1321.845444081 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26870691 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2620695 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3576404 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227092763034720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51739809029632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230152869168640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1730217736704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3576676 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚228422634392576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12069620 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184¯ñkì&ÜæM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚228422651431936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26860532 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚228820001742938 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14790912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230152869168640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1729876305408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14790912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12069622 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚228422651431936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12069622 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14801069 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6197099 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1727255747578 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚1499 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1321.845444081 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26870691 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2620695 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3576404 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚227092763034720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51739809029632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230152869168640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1730217736704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3576676 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚228422634392576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12069620 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845081482697062e+09yz¦ãA2Ý- _wandbruntime‚1321  _runtime‚1321‘«ÈÞ‚ %loss/total_loss‚9.941979638354464 trainer/global_step‚1499 % _timestamp‚1.7845081482706952e+09 2loss/SRqqt2/assignment_loss‚0.6579772342517934 /loss/FRt1/detection_loss‚0.5366753598672429 /loss/FRt2/detection_loss‚1.4838157134762997 epoch‚3 _runtime‚1321.845962933 _step‚122 2loss/SRqqt1/assignment_loss‚0.6832006947268824 1loss/SRqqt2/detection_loss‚0.7127658059034335 /loss/FBt1/detection_loss‚0.6817427652125649 /loss/FBt2/detection_loss‚0.5433942624181509 /loss/FRt1/assignment_loss‚2.343960914375179 /loss/FRt2/assignment_loss‚1.107106382638935 2loss/FBt1/assignment_loss‚0.014242341872445942 3loss/FBt2/assignment_loss‚0.0033375830740624223 1loss/SRqqt1/detection_loss‚1.1737605805374738:¦þä ߆ 1loss/SRqqt2/detection_loss‚0.7127658059034335 /loss/FBt1/detection_loss‚0.6817427652125649 /loss/FBt2/detection_loss‚0.5433942624181509 /loss/FRt1/assignment_loss‚2.343960914375179 /loss/FRt2/assignment_loss‚1.107106382638935 2loss/FBt1/assignment_loss‚0.014242341872445942 3loss/FBt2/assignment_loss‚0.0033375830740624223 1loss/SRqqt1/detection_loss‚1.1737605805374738 %loss/total_loss‚9.941979638354464 trainer/global_step‚1499 % _timestamp‚1.7845081482706952e+09 2loss/SRqqt2/assignment_loss‚0.6579772342517934 /loss/FRt1/detection_loss‚0.5366753598672429 /loss/FRt2/detection_loss‚1.4838157134762997 epoch‚3 _runtime‚1321.845962933 _step‚122 2loss/SRqqt1/assignment_loss‚0.6832006947268824zP&>\à‚(:Ñ ÞõÒ—ÿÇË disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚3282.6435546875 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚3282.6435546875 cpu‚25.588046484058783 network.sent‚ 2.198854e+06) disk./.usagePercent‚75.93996004180087( disk.nvme0c0n1.out‚10210.28857421875 disk.md0.in‚0" disk.dm-1.out‚2096.62548828125# proc.memory.rssMB‚ 2939.87890625* proc.memory.percent‚17.943596839904785 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 0.00390625 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚11302.0439453125' proc.memory.availableMB‚ 244.9921875 network.recv‚383947 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚10210.28857421875 disk.md0.out‚ 44.33984375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.50468635559082$ disk./.usageGB‚1165.868236541748ÉòI¥žá‚(:“ ÞõÒÿüË gpu.0.gpu‚52.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.432' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.81066666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ4µq^â‚(:Ê ÞõÒ䬟Ë disk.nvme1n1.in‚0 disk.dm-1.out‚2096.888671875 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme0n1.out‚10220.26953125' disk.nvme1c1n1.out‚3331.28662109375# proc.memory.rssMB‚ 2939.87890625 network.sent‚ 2.259136e+06 network.recv‚387346% disk.nvme1n1.out‚3331.28662109375# disk.dm-0.outÁ¼€÷‚11359.73876953125 disk.dm-1.in‚0% disk.nvme0c0n1.out‚10220.26953125$ memory_percent‚98.50475788116455( proc.memory.availableMB‚ 244.98046875% disk./.usageGB‚1166.1157341003418 disk.dm-0.in‚ 0.00390625 disk.nvme1c1n1.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚75.9560810356842 disk.md0.in‚0 disk.md0.out‚44.65625 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625* proc.memory.percent‚17.943596839904785 cpu‚25.586909227305963Ì‚·ã‚(:’ ÞõÒð éË gpu.0.gpu‚12.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.538' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.17933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®P¸O2ä- _wandbruntime‚1363  _runtime‚1363–ýÈÅå¿ trainer/global_step‚1549 % _timestamp‚1.7845081901617606e+09 _runtime‚1363.737317061 _step‚123 % lr-AdamW/pg1‚0.000609293600568791 % lr-AdamW/pg2‚0.000609293600568791ûhjÉæà % lr-AdamW/pg1‚0.000609293600568791 % lr-AdamW/pg2‚0.000609293600568791 trainer/global_step‚1549 % _timestamp‚1.7845081901617606e+09 _runtime‚1363.737317061 _step‚123{Ú'!2ç- _wandbruntime‚1363  _runtime‚1363.yïTÞ'èØO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235688102036992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚237472420305920 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1781971134008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235688102036992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12451479 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12451477 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12451477 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚237472420305920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2705664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51625384583680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1795070861312 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚236098836874856 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3688618 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1784676732416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1363.738141373 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1784335308288 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27727530 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1784335308288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15276051 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15276051 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234316848701488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1784676732416 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6394556 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1781630572734 _step‚124 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27717385 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15265908 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2705661 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845081901626651e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12451479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚1549 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6394279 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234316865740848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235688084997632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚236098479274222 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235688084997632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3688892 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51625403195904 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3688618 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1784676732416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1363.738141373 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1784335308288 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27727530 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1784335308288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15276051 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15276051 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234316848701488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1784676732416 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6394556 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1781630572734 _step‚124 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27717385 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15265908 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2705661 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845081901626651e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12451479 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚1549 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6394279 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234316865740848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚235688084997632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚236098479274222 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235688084997632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3688892 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51625403195904 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚237472420305920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2705664 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51625384583680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0|hàM2ê- _wandbruntime‚1364  _runtime‚1364 I…wé&ëãM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15275564 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚237622163377664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53454649088000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚237622163377664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15285721 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235836359499776 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1785445407232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15285721 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚234464296408320 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27744993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚1549 _step‚125 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 % _timestamp‚1.7845081907817688e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6398546 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6398262 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3691148 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27734834 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53454768768000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2707395 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12459270 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27734834 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚277449936§­•RìçM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27734834 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53454768768000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2707395 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_ba#X°®›#tch_end/active.large_pool.freed‚12459270 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27734834 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27744993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15275564 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚237622163377664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53454649088000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚237622163377664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15285721 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235836359499776 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1785445407232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15285721 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚234464296408320 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27744993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚1549 _step‚125 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 % _timestamp‚1.7845081907817688e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6398546 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6398262 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3691148 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776}“!'2í- _wandbruntime‚1364  _runtime‚1364õž^î‚ 3loss/FBt2/assignment_loss‚0.0035723100995449877 1loss/SRqqt2/detection_loss‚0.8473367689871829 /loss/FBt1/detection_loss‚0.5253887444730378 _runtime‚1364.358089475 trainer/global_step‚1549 % _timestamp‚1.7845081907828126e+09 /loss/FRt1/assignment_loss‚2.065458407255114 0loss/FRt2/assignment_loss‚1.2188065272790414 2loss/FBt1/assignment_loss‚0.049509493864203814 /loss/FBt2/detection_loss‚0.7450631786681091 _step‚126 2loss/SRqqt1/assignment_loss‚0.6337802751908511 /loss/FRt1/detection_loss‚0.5066862777977148 /loss/FRt2/detection_loss‚1.6553091158607491 1loss/SRqqt1/detection_loss‚0.7624930140100286 $loss/total_loss‚9.65526903122214 epoch‚3 2loss/SRqqt2/assignment_loss‚0.6418649177365625S¾« ï† /loss/FRt2/detection_loss‚1.6553091158607491 1loss/SRqqt1/detection_loss‚0.7624930140100286 $loss/total_loss‚9.65526903122214 epoch‚3 2loss/SRqqt2/assignment_loss‚0.6418649177365625 3loss/FBt2/assignment_loss‚0.0035723100995449877 1loss/SRqqt2/detection_loss‚0.8473367689871829 /loss/FBt1/detection_loss‚0.5253887444730378 _runtime‚1364.358089475 trainer/global_step‚1549 % _timestamp‚1.7845081907828126e+09 /loss/FRt1/assignment_loss‚2.065458407255114 0loss/FRt2/assignment_loss‚1.2188065272790414 2loss/FBt1/assignment_loss‚0.049509493864203814 /loss/FBt2/detection_loss‚0.7450631786681091 _step‚126 2loss/SRqqt1/assignment_loss‚0.6337802751908511 /loss/FRt1/detection_loss‚0.5066862777977148~ïMe1Vð‚(:Ë ŸÞõÒ»‘ÌË disk.nvme0n1.in‚ 0.00390625 disk.dm-0.in‚ 0.00390625* proc.memory.percent‚17.943596839904785 proc.cpu.threads‚17( disk./.usagePercent‚75.9561168159419 disk.nvme1c1n1.in‚0 disk.md0.out‚ 44.953125# disk.dm-0.out‚11360.49658203125 disk.nvme0c0n1.in‚ 0.00390625 cpu‚25.585589900515348$ disk./.usageGB‚1166.116283416748 disk.nvme1n1.in‚0% disk.nvme1n1.out‚3331.92919921875 disk.dm-1.in‚0 disk.dm-1.out‚2096.994140625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚10221.150390625$ memory_percent‚98.50506782531738 network.sent‚ 2.263984e+06 network.recv‚390868$ disk.nvme0n1.out‚10221.150390625' disk.nvme1c1n1.out‚3331.92919921875 disk.md0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 244.9296875# proc.memory.rssMB‚ 2939.87890625yIó­žñ‚(:“ ŸÞõÒûÐÌ gpu.0.gpu‚92.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚266.919' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.97299999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P‹­N\ò‚(:Ñ ®ÞõÒÙÚ­Ë disk.nvme1n1.in‚0$ disk.nvme1n1.out‚3358.6259765625$ memory_percent‚98.50401878356934* proc.memory.percent‚17.943596839904785 network.sent‚ 2.328292e+06 network.recv‚394769% disk./.usageGB‚1166.3183975219727 disk.md0.in‚0 disk.md0.out‚ 45.55078125' disk.nvme0c0n1.out‚10429.1826171875% disk.nvme0n1.out‚10429.1826171875' proc.memory.availableMB‚ 245.1015625) disk./.usagePercent‚75.96928171450725 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625& disk.nvme1c1n1.out‚3358.6259765625# proc.memory.rssMB‚ 2939.87890625 proc.cpu.threads‚17" disk.dm-1.out‚2097.18505859375 disk.nvme0c0n1.in‚ 0.00390625 cpu‚25.584321283100785 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚11593.77294921875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0j6ŽØžó‚(:“ ®ÞõÒÊ’öË gpu.0.gpu‚71.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚226.441' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.48033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 U¶_ô‚(:Ô ½ÞõÒß·£Ë( proc.memory.availableMB‚ 245.07421875) disk./.usagePercent‚75.97161910953592( disk.nvme0c0n1.out‚10457.60400390625 disk.nvme1c1n1.in‚0* proc.memory.percent‚17.943596839904785 cpu‚25.582719542589363 proc.cpu.threads‚17 network.sent‚ 2.331256e+06% disk./.usageGB‚1166.3542823791504 disk.rbd0.out‚0# proc.memory.rssMB‚ 2939.87890625 network.recv‚398499 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚10457.60400390625' disk.nvme1c1n1.out‚3359.33642578125 disk.nvme1n1.in‚0# disk.dm-0.out‚11621.74560546875 disk.dm-1.in‚0" disk.dm-1.out‚2097.28271484375% disk.nvme1n1.out‚3359.33642578125 disk.md0.in‚0 disk.md0.out‚ 46.0546875# memory_percent‚98.5041856765747‡}hìžõ‚(:“ ½ÞõÒõæïË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚275.984' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.99466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡z׉2ö- _wandbruntime‚1406  _runtime‚1406ÐA÷ðÇ÷Á _runtime‚1406.561042657 _step‚127 & lr-AdamW/pg1‚0.0006088329804782238 & lr-AdamW/pg2‚0.0006088329804782238 trainer/global_step‚1599 % _timestamp‚1.7845082329841397e+09V(yËøÅ % _timestamp‚1.7845082329841397e+09 _runtime‚1406.561042657 _step‚127 & lr-AdamW/pg1‚0.0006088329804782238 & lr-AdamW/pg2‚0.0006088329804782238 trainer/global_step‚1599DÖZ22ù- _wandbruntime‚1406  _runtime‚1406[ÈyhÞ'úØO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1837454347416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15760703 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2792361 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1839904410112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12841127 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1840245834240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1850890875392 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15760703 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6595312 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚241688399114448 _step‚128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243101810104832 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1406.561137457 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12841129 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243525853461864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12841129 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15750560 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6595602 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244942072978432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚1599 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53284398087680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚241688382075088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1839904410112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244941714514944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2792364 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243101810104832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244941714514944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55135307575296 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28591687 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1850793994752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344¾|4ÅðûÝO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚1599 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53284398087680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚241688382075088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1839904410112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244941714514944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2792364 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243101810104832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244941714514944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55135307575296 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28591687 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1850793994752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1837454347416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15760703 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2792361 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1839904410112 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12841127 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1840245834240 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1850890875392 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15760703 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6595312 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚241688399114448 _step‚128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243101810104832 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatWö®ó sMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1406.561137457 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12841129 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243525853461864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12841129 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15750560 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6595602 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244942072978432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287€D‘ÿ2ü- _wandbruntime‚1407  _runtime‚1407õ}v¡è&ýâM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1841355940352 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3805205 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15770373 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12848922 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1841014509056 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28609136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12848920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53317578357760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243250084606976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245091457586688 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28619295 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92679168 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚1407.179583511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6599568 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55169605617152 trainer/global_step‚1599 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1838562735780 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚129 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28619295 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚241835812742560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245091457586688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111291392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1851915968000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1841355940352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845082336038296e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2794095 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3805470 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55169494325760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243250101646336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243674392517700 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15760216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15770373 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚241835829781920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243250084606976 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6599300 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28609136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1841014509056 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1838222174394 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0«p ˆí&þçM FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1838222174394 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1841355940352 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3805205 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15770373 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12848922 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1841014509056 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28609136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12848920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53317578357760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243250084606976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245091457586688 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28619295 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92679168 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚1407.179583511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6599568 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55169605617152 trainer/global_step‚1599 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1838562735780 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚129 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28619295 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚241835812742560 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245091457586688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111291392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1851915968000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1841355940352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845082336038296e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2794095 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3805470 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55169494325760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚243250101646336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243674392517700 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15760216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15770373 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚241835829781920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚243250084606976 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6599300 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28609136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1841014509056|3µ2ÿ- _wandbruntime‚1407  _runtime‚1407µÏ¼€  2loss/FBt2/assignment_loss‚0.012824052067956542 1loss/SRqqt1/detection_loss‚0.7470852715434667 0loss/SRqqt2/detection_loss‚0.876166801659266 0loss/FBt1/detection_loss‚0.40719591118567794 %loss/total_loss‚9.149006805835329 epoch‚3 % _timestamp‚1.7845082336048577e+09 .loss/FRt1/assignment_loss‚2.01437053051321 2loss/SRqqt1/assignment_loss‚0.5722699607710964 /loss/FBt2/detection_loss‚0.6979057820370564 _step‚130 2loss/SRqqt2/assignment_loss‚0.5772449149268665 2loss/FBt1/assignment_loss‚0.043727809121131674 _runtime‚1407.180142022 0loss/FRt2/assignment_loss‚1.0547321773123468 /loss/FRt1/detection_loss‚0.5659095301486665 /loss/FRt2/detection_loss‚1.5795740645485876 trainer/global_step‚1599ͳ  † _step‚130 2loss/SRqqt2/assignment_loss‚0.5772449149268665 2loss/FBt1/assignment_loss‚0.043727809121131674 _runtime‚1407.180142022 0loss/FRt2/assignment_loss‚1.0547321773123468 /loss/FRt1/detection_loss‚0.5659095301486665 /loss/FRt2/detection_loss‚1.5795740645485876 trainer/global_step‚1599 2loss/FBt2/assignment_loss‚0.012824052067956542 1loss/SRqqt1/detection_loss‚0.7470852715434667 0loss/SRqqt2/detection_loss‚0.876166801659266 0loss/FBt1/detection_loss‚0.40719591118567794 %loss/total_loss‚9.149006805835329 epoch‚3 % _timestamp‚1.7845082336048577e+09 .loss/FRt1/assignment_loss‚2.01437053051321 2loss/SRqqt1/assignment_loss‚0.5722699607710964 /loss/FBt2/detection_loss‚0.6979057820370564‚2ÞÅ U‚ ‚(:Ê ÌÞõÒ»ÆáË% disk.nvme1n1.out‚3425.27001953125 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚11820.20947265625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2941.19921875* proc.memory.percent‚17.951655387878418 cpu‚25.585512428997472 network.recv‚401508 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚3425.27001953125 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 46.37890625 disk.dm-1.in‚0 disk.rbd0.out‚0% disk./.usageGB‚1166.2901039123535! disk.dm-1.out‚2097.3232421875 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme0c0n1.out‚ 10590.8828125$ memory_percent‚98.51248264312744 proc.cpu.threads‚17 network.sent‚ 2.34998e+06' disk./.usagePercent‚75.967438782762 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 243.71484375" disk.nvme0n1.out‚ 10590.8828125Æ®„Šƒ ‚(:’ ÌÞõÒë¦Ì gpu.0.gpu‚15.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.486' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.49533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„h2„ - _wandbruntime‚1423  _runtime‚1423ÞÁ‘ßP… Ò{ @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5836111307144165 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 )loss/val_total_loss‚9.088569201355172 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012232686392962933 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5528180003166199 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7916666865348816 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6250582933425903 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12417510896921158 /particle/accuracy_4_of_4‚0.9471108317375183 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4632139205932617 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974045753479004 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_3‚0.4670500159263611 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.996985137462616 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.10547668486833572 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4722222089767456 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939106702804565 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5715839862823486 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_1_of_4‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11653956770896912 *jet/accuracy_1_of_3‚0.8983613848686218 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6279661655426025 particle/accuracy_2_of_5‚1 epoch‚3 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5809491872787476 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12258241325616837 *jet/accuracy_2_of_4‚0.7708700895309448 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9852544665336609 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8388888835906982 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.027850758284330368 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12685149908065796 7validation_average_jet_accuracy‚0.31844568252563477 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6327908635139465 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5940660834312439 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4510182738304138 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10974548757076263 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975579380989075 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9895392060279846 +particle/sensitivity‚0.6106560826301575 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9389649629592896 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977214336395264 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11833633482456207 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974045753479004 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8600718379020691 (particle/accuracy‚0.8446296453475952 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5821905732154846 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5798737406730652 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0076730335131287575 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12360256910324097 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12028631567955017 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8600718379020691 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.338309645652771 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6308573484420776 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6356421709060669 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20256245136260986 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.10547668486833572 +jet/accuracy_3_of_4‚0.09023622423410416 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6330686211585999 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7829176783561707 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5573865175247192 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6830622553825378 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3489583432674408 jet/accuracy_4_of_5‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1317194253206253 particle/accuracy_2_of_2‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6396277546882629 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7261978983879089 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7845082501319265e+09 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8445578217506409 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981421232223511 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.731732964515686 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12942945957183838 +jet/accuracy_2_of_2‚0.12543562054634094 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/*FRt*SRqqt*FBt/SRqqt_ð•ŽËˆ"purity‚0.6299117207527161 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.574985146522522 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7108461260795593 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13725876808166504 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7229393124580383 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.10561611503362656 +jet/accuracy_1_of_1‚0.26689064502716064 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6468347907066345 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1417263299226761 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9937848448753357 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997259259223938 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9224797487258911 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7313371896743774 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14916904270648956 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7486644983291626 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5842607617378235 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3256731331348419 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958823323249817 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12514859437942505 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 particle/accuracy_3_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.341838538646698 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5842607617378235 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5755448341369629 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9862618446350098 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13505932688713074 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.695665180683136 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004276742693036795 *jet/accuracy_1_of_4‚0.9602829217910767 trainer/global_step‚1599 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.613625168800354 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022848710417747498 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12520143389701843 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3162335753440857 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01600479707121849 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +particle/specificity‚0.9086220860481262 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9900732636451721 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.682981014251709 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_2‚0.7548615336418152 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8768922090530396 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08002789318561554 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18271881341934204 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4242216646671295 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.69508296251297 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44212964177131653 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13231855630874634 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29262441396713257 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7486644983291626 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8543494939804077 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47684338688850403 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12436602264642715 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7326831221580505 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7356179356575012 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.620429515838623 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1317194253206253 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9947417378425598 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24486573040485382 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.14447861909866333 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 _runtime‚1423.707944328 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.991936445236206 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6402323246002197 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14249999821186066 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8948310017585754 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912295341491699 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.557161808013916 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10637107491493225 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 _step‚131 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.624028742313385 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1179397702217102 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_3_of_3‚0.07195848971605301 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1216958612203598 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10126849263906479 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09333854913711548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5349999666213989 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5302573442459106 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10872042924165726 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6137757301330566 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9950254559516907 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8579743504524231 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1208517923951149 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6432827711105347 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9801465272903442 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_2_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ùeÝ=† ×{ 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1317194253206253 particle/accuracy_2_of_2‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6396277546882629 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7261978983879089 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 % _timestamp‚1.7845082501319265e+09 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8445578217506409 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981421232223511 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.731732964515686 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12942945957183838 +jet/accuracy_2_of_2‚0.12543562054634094 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6299117207527161 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.574985146522522 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7108461260795593 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13725876808166504 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7229393124580383 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.10561611503362656 +jet/accuracy_1_of_1‚0.26689064502716064 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6468347907066345 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1417263299226761 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9937848448753357 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997259259223938 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9224797487258911 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7313371896743774 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.14916904270648956 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7486644983291626 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5842607617378235 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3256731331348419 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958823323249817 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12514859437942505 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 particle/accuracy_3_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.341838538646698 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5842607617378235 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5755448341369629 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9862618446350098 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13505932688713074 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.695665180683136 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004276742693036795 *jet/accuracy_1_of_4‚0.9602829217910767 trainer/global_step‚1599 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.613625168800354 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022848710417747498 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12520143389701843 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3162335753440857 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01600479707121849 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +particle/specificity‚0.9086220860481262 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9900732636451721 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.682981014251709 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_2‚0.7548615336418152 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8768922090530396 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08002789318561554 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18271881341934204 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4242216646671295 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.69508296251297 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44212964177131653 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13231855630874634 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29262441396713257 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7486644983291626 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8543494939804077 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47684338688850403 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12436602264642715 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7326831221580505 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7356179356575012 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.620429515838623 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1317194253206253 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9947417378425598 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24486573040485382 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.14447861909866333 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 _runtime‚1423.707944328 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.991936445236206 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961110949516296 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6402323246002197 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14249999821186066 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8948310017585754 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912295341491699 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.557161808013916 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10637107491493225 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 _step‚131 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.624028742313385 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1179397702217102 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_3_of_3‚0.07195848971605301 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1216958612203598 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10126849263906479 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09333854913711548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5349999666213989 particle/accuracy_2_of_3‚1 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5302573442459106 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10872042924165726 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6137757301330566 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9950254559516907 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8579743504524231 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1208517923951149 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6432827711105347 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9801465272903442 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_2_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5836111307144165 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 )loss/val_total_loss‚9.088569201355172 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012232686392962933 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5528180003166199 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7916666865348816 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6250582933425903 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12417510896921158 /particle/accuracy_4_of_4‚0.9471108317375183 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4632139205932617 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974045753479004 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_3‚0.4670500159263611 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.996985137462616 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.10547668486833572 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4722222089767456 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939106702804565 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5715839862823486 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 particle/accuracy_1_of_4‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11653956770896912 *jet/accuracy_1_of_3‚0.8983613848686218 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6279661655426025 particle/accuracy_2_of_5‚1 epoch‚3 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5809491872787476 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12258241325616837 *jet/accuracy_2_of_4‚0.7708700895309448 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9852544665336609 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8388888835906982 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.027850758284330368 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12685149908065796 7validation_average_jet_accuracy‚0.31844568252563477 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6327908635139465 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5940660834312439 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4510182738304138 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10974548757076263 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975579380989075 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9895392060279846 +particle/sensitivity‚0.6106560826301575 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9389649629592896 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977214336395264 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11833633482456207 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974045753479004 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8600718379020691 (particle/accuracy‚0.8446296453475952 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5821905732154846 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5798737406730652 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0076730335131287575 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12360256910324097 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12028631567955017 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8600718379020691 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.338309645652771 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6308573484420776 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6356421709060669 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20256245136260986 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.10547668486833572 +jet/accuracy_3_of_4‚0.09023622423410416 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6330686211585999 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7829176783561707 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5573865175247192 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6830622553825378 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3489583432674408 jet/accuracy_4_of_5‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505ƒØR‡ ‚(:Ç ÛÞõÒËí¢Ë network.sent‚ 2.398479e+06 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚11820.73681640625% disk.nvme1n1.out‚3671.20361328125& proc.memory.availableMB‚ 740.078125# proc.memory.rssMB‚ 2958.77734375 cpu‚25.608058332516435 proc.cpu.threads‚17 disk.rbd0.in‚0% disk.nvme0c0n1.out‚10591.76953125' disk.nvme1c1n1.out‚3671.20361328125 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚46.78125! disk.dm-1.out‚2342.7646484375 disk.nvme0n1.in‚ 0.00390625# memory_percent‚95.4829216003418) proc.memory.percent‚18.05894374847412 network.recv‚405619) disk./.usagePercent‚75.96744300682019$ disk./.usageGB‚1166.290168762207 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625# disk.nvme0n1.out‚10591.76953125 disk.nvme1c1n1.in‚0-¨ô—‘ˆ ‚(:† ÛÞõÒŽ–‹Ì gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.062' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏÄ‚2\‰ ‚(:Ñ êÞõÒ¼Ú²Ë proc.cpu.threads‚17 network.sent‚ 2.457281e+06 network.recv‚409610 disk.nvme0c0n1.in‚ 0.00390625) disk./.usagePercent‚75.96740250555628% disk./.usageGB‚1166.2895469665527 disk.rbd0.out‚0& disk.nvme0n1.out‚10592.94091796875 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625" proc.memory.rssMB‚ 2959.3515625 disk.dm-1.in‚0" disk.dm-1.out‚2342.82568359375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚10592.94091796875 disk.nvme1n1.in‚0 disk.md0.out‚ 47.05078125( proc.memory.availableMB‚ 739.37109375 cpu‚25.606597286334626 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚3671.7021484375$ disk.nvme1n1.out‚3671.7021484375! disk.dm-0.out‚11821.751953125$ memory_percent‚95.48723697662354* proc.memory.percent‚18.062448501586914Uó×.“Š ‚(:ˆ êÞõÒÚ—õË gpu.0.gpu‚49.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚242.529' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚80.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lцV‹ ‚(:Ë ùÞõÒ¨Ž·Ë disk.nvme1n1.in‚0$ disk.nvme1n1.out‚3917.6083984375 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚3917.6083984375$ memory_percent‚95.48876285552979( proc.memory.availableMB‚ 739.12109375) disk./.usagePercent‚75.96745046104054$ disk./.usageGB‚1166.290283203125 disk.md0.in‚0 disk.md0.out‚ 47.4140625" disk.dm-1.out‚2588.25927734375* proc.memory.percent‚18.064498901367188 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625! disk.dm-0.out‚11822.509765625 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚10593.99951171875 disk.rbd0.out‚0 network.sent‚ 2.460245e+06 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚10593.99951171875 proc.memory.rssMB‚ 2959.6875 cpu‚25.60482950850568 network.recv‚413234¦§m±“Œ ‚(:ˆ ùÞõÒ·­ŽÌ gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚252.852' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚84.284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v<Â2 - _wandbruntime‚1467  _runtime‚1467³q+ÇŽ Á & lr-AdamW/pg1‚0.0006083529899366678 & lr-AdamW/pg2‚0.0006083529899366678 trainer/global_step‚1649 % _timestamp‚1.7845082940523024e+09 _runtime‚1467.627916921 _step‚132î"þbÌ Æ & lr-AdamW/pg2‚0.0006083529899366678 trainer/global_step‚1649 % _timestamp‚1.7845082940523024e+09 _runtime‚1467.627916921 _step‚132 & lr-AdamW/pg1‚0.0006083529899366678„‰1v2 - _wandbruntime‚1467  _runtime‚1467ó=0Ž‘ ÙO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3956157 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56893289791488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252457879087616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13353585 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚252894429905998 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16316711 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845082940533776e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1915791350272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚254373670437888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56893415770112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29670296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚254373670437888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1927406903808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13353585 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚254374028901376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚1649 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _runtime‚1467.632050509 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1927299537408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚133 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54966008866304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1916132774400 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6839337 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54965990254080 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3955880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚250981188496272èüLDä'’ ÞO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3956157 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56893289791488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252457879087616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13353585 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚252894429905998 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16316711 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845082940533776e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1915791350272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚254373670437888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56893415770112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29670296 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚254373670437888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1927406903808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13353585 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚254374028901376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚1649 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _runtime‚1467.632050509 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1927299537408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚133 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54966008866304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1916132774400 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6839337 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54965990254080 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3955880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚250981188496272…MˆÐ2“ - _wandbruntime‚1468  _runtime‚1468f?Çé&” ãM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13361378 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚254523413509632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1917242880512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13361380 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16326367 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16326367 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚254523413509632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29687745 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253042968961834 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3958189 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13361380 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚1649 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13361378 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252606153589760 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56927584620032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251128602124384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1914009236704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1468.249177323 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6843100 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3958479 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29697904 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29697904 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16336524 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360ìf£H• èM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚134 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2884914 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252606153589760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1916901449216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16336524 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13361378 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚254523413509632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1917242880512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13361380 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16326367 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16326367 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚254523413509632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29687745 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253042968961834 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3958189 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13361380 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚1649 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13361378 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252606153589760 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56927584620032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251128602124384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1914009236704 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1468.249177323 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6843100 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3958479 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 Ba_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56927706397184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚253042611361088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1914349798090 % _timestamp‚1.7845082946734495e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29697904 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29697904 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 Fî;@ ˜ „ % _timestamp‚1.7845082946745028e+09 _runtime‚ 1468.25300559 2loss/SRqqt2/assignment_loss‚0.6610094480080027 1loss/FBt1/assignment_loss‚0.01511482100764223 /loss/FRt1/detection_loss‚0.5520584891571734 /loss/FRt2/detection_loss‚1.6116537020813824 1loss/SRqqt1/detection_loss‚0.7521592414320827 1loss/SRqqt2/detection_loss‚0.8103117212963074 epoch‚4 trainer/global_step‚1649 2loss/SRqqt1/assignment_loss‚0.5680226527903318 /loss/FBt2/detection_loss‚0.8209379298302034 %loss/total_loss‚9.403322243929768 /loss/FRt1/assignment_loss‚2.034750935826468 0loss/FRt2/assignment_loss‚0.9965714413026733 0loss/FBt2/assignment_loss‚0.0973597171385096 0loss/FBt1/detection_loss‚0.48337214405899154 _step‚135‡YYw|W™ ‚(:Ì ˆßõÒŸø“Ë disk.nvme1n1.in‚0$ disk.nvme1n1.out‚3920.5498046875) disk./.usagePercent‚75.96745294578066% disk./.usageGB‚1166.2903213500977' disk.nvme0c0n1.out‚10597.6708984375 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚2590.47021484375 disk.nvme0n1.in‚ 0.00390625& disk.nvme1c1n1.out‚3920.5498046875# disk.dm-0.out‚11826.59912109375* proc.memory.percent‚18.064498901367188 cpu‚25.603108559618217 proc.cpu.threads‚17 network.sent‚ 2.466111e+06 network.recv‚416326 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.md0.in‚0 disk.md0.out‚ 47.55078125 disk.dm-1.in‚0% disk.nvme0n1.out‚10597.6708984375$ memory_percent‚95.49055099487305& proc.memory.availableMB‚ 738.828125 proc.memory.rssMB‚ 2959.6875 disk.nvme1c1n1.in‚0MÂ实š ‚(:“ ˆßõÒÀÇ„Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.884' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.96133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýÜË—V› ‚(:Ë —ßõÒ÷†¸Ë disk.md0.out‚ 48.203125 disk.dm-0.in‚ 0.00390625( disk./.usagePercent‚75.9674705874355& disk.nvme1c1n1.out‚3995.7021484375 disk.nvme1n1.in‚0 disk.md0.in‚0# disk.dm-0.out‚11827.46240234375" disk.dm-1.out‚2664.82568359375( proc.memory.availableMB‚ 739.05078125 cpu‚25.601735119504987' disk.nvme0c0n1.out‚10599.1357421875% disk.nvme0n1.out‚10599.1357421875 disk.dm-1.in‚0 proc.memory.rssMB‚ 2959.6875 network.sent‚ 2.528423e+06 network.recv‚420321 disk.rbd0.in‚0$ memory_percent‚95.48919200897217* proc.memory.percent‚18.064498901367188 proc.cpu.threads‚17% disk./.usageGB‚1166.2905921936035 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚3995.7021484375ÑïÞC‘œ ‚(:† —ßõÒÌ gpu.0.gpu‚4.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.958' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È-SKž ‚(:“ ¦ßõÒŠâÑË gpu.0.gpu‚93.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚276.241' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.08033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,0-Sž ‚(:È ¦ßõÒßÎÌ" disk.dm-1.out‚2666.93408203125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625 disk.md0.in‚0 cpu‚25.603998276928102 disk.dm-1.in‚0& disk.nvme0n1.out‚10600.28173828125 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625% proc.memory.availableMB‚ 739.03125 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 3998.29296875 network.recv‚423606) disk./.usagePercent‚75.96746686032533 disk.nvme0c0n1.in‚ 0.00390625 disk.md0.out‚ 48.54296875" disk.dm-0.out‚11828.3603515625$ memory_percent‚95.48931121826172 network.sent‚ 2.534055e+06( disk.nvme0c0n1.out‚10600.28173828125" disk.nvme1n1.out‚ 3998.29296875 proc.memory.rssMB‚ 2959.6875* proc.memory.percent‚18.064498901367188% disk./.usageGB‚1166.29053497314454•Ð‚2Ÿ - _wandbruntime‚1510  _runtime‚1510§²ù]Ç  Á % _timestamp‚1.7845083372138386e+09 _runtime‚1510.789443818 _step‚136 & lr-AdamW/pg1‚0.0006078536597730185 & lr-AdamW/pg2‚0.0006078536597730185 trainer/global_step‚1699ê8ƒÌ¡ Æ & lr-AdamW/pg1‚0.0006078536597730185 & lr-AdamW/pg2‚0.0006078536597730185 trainer/global_step‚1699 % _timestamp‚1.7845083372138386e+09 _runtime‚1510.789443818 _step‚136ˆp-Ž2¢ - _wandbruntime‚1510  _runtime‚1510àØ“¸ß'£ ÙO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30544598 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259871621234176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚261842964646912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58608132994048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4070902 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16811506 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261842964646912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚260321446493006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2969880 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2969877 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56625003758080 trainer/global_step‚1699 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1968724623134 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261843323110400 _runtime‚1510.790381639 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1971701876224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258352721869872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1968384061860 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16801363 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260321088892372 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13743235 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258352704830512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7040495 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1971360452096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1971360452096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1983129235968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845083372148583e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16811506 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚259871621234176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13743237 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56625022370304 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0=!|ßä'¤ ÞO PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7040782 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1983238699520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259871604194816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13743235 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1971360452096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1983129235968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845083372148583e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16811506 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚259871621234176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13743237 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56625022370304 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30544598 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚259871621234176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚261842964646912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58608132994048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4070902 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16811506 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261842964646912 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚260321446493006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2969880 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2969877 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56625003758080 trainer/global_step‚1699 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1968724623134 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261843323110400 _runtime‚1510.790381639 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1971701876224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258352721869872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1968384061860 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16801363 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260321088892372 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13743235 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258352704830512 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7040495 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1971360452096 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_poÞ÷<Áol.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1972470551040 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16821176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4072864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56658202640384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚260469627948096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260019895736320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261992707718656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260019878696960 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30562047 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7044769 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1972470551040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.784508337823883e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚1511.399592449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13751028 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258500135497984 _step‚138 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13751028 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1969492450112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30572206 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16811019 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258500152537344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7044475 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2971611 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1984354000384 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13751030 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16811019 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4073155 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1969833011498 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2971614t*µí&§ çM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260019895736320 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1972811982336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1984246641152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1972470551040 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16821176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4072864 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚56658202640384 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚260469627948096 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260019895736320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261992707718656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260019878696960 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30562047 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7044769 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1972470551040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.784508337823883e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚1511.399592449 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13751028 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258500135497984 _step‚138 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13751028 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1969492450112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30572206 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16811019 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258500152537344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7044475 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2971611 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1984354000384 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13751030 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16811019 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4073155 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1969833011498 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2971614ŠÙ=kø2¨ - _wandbruntime‚1511  _runtime‚1511›!© þ /loss/FBt2/detection_loss‚0.6098806247210655 /loss/FRt1/assignment_loss‚2.017046653233799 /loss/FRt2/assignment_loss‚0.920161782829294 2loss/SRqqt2/assignment_loss‚0.6087177553443619 2loss/FBt2/assignment_loss‚0.001733495443215925 /loss/FRt2/detection_loss‚1.7450796973679585 0loss/SRqqt2/detection_loss‚0.717381129106391 $loss/total_loss‚9.12142449146351 epoch‚4 trainer/global_step‚1699 # _timestamp‚1.78450833782686e+09 _runtime‚1511.402164764 2loss/SRqqt1/assignment_loss‚0.5872007993424673 3loss/FBt1/assignment_loss‚0.0023428851144735533 /loss/FRt1/detection_loss‚0.5396131664518719 _step‚139 1loss/SRqqt1/detection_loss‚0.8176510699284315 /loss/FBt1/detection_loss‚0.5546154325801785§u^ ª ƒ _step‚139 1loss/SRqqt1/detection_loss‚0.8176510699284315 /loss/FBt1/detection_loss‚0.5546154325801785 /loss/FBt2/detection_loss‚0.6098806247210655 /loss/FRt1/assignment_loss‚2.017046653233799 /loss/FRt2/assignment_loss‚0.920161782829294 2loss/SRqqt2/assignment_loss‚0.6087177553443619 2loss/FBt2/assignment_loss‚0.001733495443215925 /loss/FRt2/detection_loss‚1.7450796973679585 0loss/SRqqt2/detection_loss‚0.717381129106391 $loss/total_loss‚9.12142449146351 epoch‚4 trainer/global_step‚1699 # _timestamp‚1.78450833782686e+09 _runtime‚1511.402164764 2loss/SRqqt1/assignment_loss‚0.5872007993424673 3loss/FBt1/assignment_loss‚0.0023428851144735533 /loss/FRt1/detection_loss‚0.5396131664518719‹}õ×¹[« ‚(:Ð µßõÒøüþË disk.md0.out‚ 49.00390625" disk.dm-1.out‚2670.19580078125* proc.memory.percent‚18.064498901367188 cpu‚25.60674314430415& disk.nvme0n1.out‚10601.41455078125 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚11829.09228515625 disk.dm-1.in‚0 disk.rbd0.out‚0 network.sent‚ 2.549815e+06 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme1n1.out‚4002.13525390625 disk.md0.in‚0 disk.rbd0.in‚0$ memory_percent‚95.48723697662354( proc.memory.availableMB‚ 739.37109375 network.recv‚427139( disk./.usagePercent‚75.9674683511694% disk./.usageGB‚1166.2905578613281( disk.nvme0c0n1.out‚10601.41455078125' disk.nvme1c1n1.out‚4002.13525390625 proc.memory.rssMB‚ 2959.6875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 ³“«œ¬ ‚(:‘ µßõÒ‹ÝÉÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.245' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.08166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÑ’ðW­ ‚(:Ì ÄßõÒѼ²Ë disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 739.3515625 proc.memory.rssMB‚ 2959.6875% disk./.usageGB‚1166.2906036376953 disk.md0.out‚ 49.3359375 disk.dm-1.in‚0" disk.dm-1.out‚2671.10107421875$ disk.nvme1n1.out‚4003.7490234375 cpu‚25.611387276558414 network.sent‚ 2.599569e+06 network.recv‚431583) disk./.usagePercent‚75.96747133285754 disk.rbd0.in‚0( disk.nvme0c0n1.out‚10603.88037109375 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚10603.88037109375& disk.nvme1c1n1.out‚4003.7490234375 disk.md0.in‚0! disk.dm-0.out‚11831.544921875 disk.nvme1c1n1.in‚0$ memory_percent‚95.48735618591309* proc.memory.percent‚18.064498901367188 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625Ú©‚à“® ‚(:ˆ ÄßõÒíŒÿË gpu.0.gpu‚93.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚267.018' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½÷ÙP¯ ‚(:Å ÓßõÒ’¼ÅË disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 proc.cpu.threads‚17 network.recv‚435141 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚2671.14794921875 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0n1.out‚ 10604.8203125' disk.nvme1c1n1.out‚4004.33349609375% disk.nvme1n1.out‚4004.33349609375! disk.dm-0.out‚11832.103515625$ disk.nvme0c0n1.out‚ 10604.8203125 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 49.76171875" memory_percent‚95.487380027771 proc.memory.rssMB‚ 2959.6875 cpu‚25.61511591139415 network.sent‚ 2.605444e+06% disk./.usageGB‚1166.2906227111816( proc.memory.availableMB‚ 739.34765625* proc.memory.percent‚18.064498901367188) disk./.usagePercent‚75.96747257522759rÖßœ° ‚(:‘ ÓßõÒÅù Ì gpu.0.gpu‚3.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.231' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.74366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶u>2± - _wandbruntime‚1553  _runtime‚1553’n@çŲ ¿ _step‚140 & lr-AdamW/pg1‚0.0006073350220583191 & lr-AdamW/pg2‚0.0006073350220583191 trainer/global_step‚1749 # _timestamp‚1.78450837967926e+09 _runtime‚1553.256067313 8b›Ê³ Ä _step‚140 & lr-AdamW/pg1‚0.0006073350220583191 & lr-AdamW/pg2‚0.0006073350220583191 trainer/global_step‚1749 # _timestamp‚1.78450837967926e+09 _runtime‚1553.256067313Œ*ì•Ã2´ - _wandbruntime‚1553  _runtime‚1553)Ñòeã#µ ØO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4185562 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3056577 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚1749 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2026929553920 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267285346341376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17286015 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58284017262080 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7241848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3056580 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17296158 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚269312617319424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60322996812288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14132887 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚267748105479380 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2039080625152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267285329302016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17296158 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60323116499456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _runtime‚1553.256134373 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267285329302016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚265724255243472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚141 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2024207836542 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269312258855936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2027270978048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ …ÅŠû171722240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2027270978048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14132885 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚267748463080014 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269312258855936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267285346341376:Òêìã'¶ ÝO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚267748105479380 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2039080625152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267285329302016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17296158 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60323116499456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _runtime‚1553.256134373 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267285329302016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚265724255243472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚141 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2024207836542 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269312258855936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2027270978048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2027270978048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14132885 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚267748463080014 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269312258855936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267285346341376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4185562 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3056577 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚1749 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2026929553920 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267285346341376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17286015 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58284017262080 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7241848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3056580 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17296158 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚269312617319424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60322996812288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14132887 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496^› ‰2· - _wandbruntime‚1553  _runtime‚1553š¿†ªé&¸ ãM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7245808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2040091548672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚267433620843520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14140678 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚267433603804160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31436349 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269461643457024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3058314 _step‚142 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚269462001927680 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 % _timestamp‚1.7845083802928374e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31436349 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267897002135850 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269462001927680 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31446508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265871668871584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4187497 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2040201005056 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265871685910944 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14140678 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2024975663520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚267896644535104 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31446508 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _runtime‚1553.868537281 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2028039652864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17305828 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60357289080832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58317216144384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2028381084160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚1749 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17305828 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚267433620843520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2028039652864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58317197532160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184o§xî&¹ èM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2040091548672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚267433620843520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14140678 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚267433603804160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31436349 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269461643457024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3058314 _step‚142 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚269462001927680 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 % _timestamp‚1.7845083802928374e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31436349 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267897002135850 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269462001927680 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31446508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265871668871584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4187497 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2040201005056 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265871685910944 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14140678 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2024975663520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚267896644535104 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31446508 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _runtime‚1553.868537281 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2028039652864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17305828 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60357289080832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58317216144384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2028381084160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚1749 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17305828 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚267433620843520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2028039652864 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58317197532160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7245808ŽÜo@÷2º - _wandbruntime‚1553  _runtime‚1553 ãk » ƒ /loss/FBt1/detection_loss‚0.4861992402260916 %loss/total_loss‚9.131391071271416 _runtime‚1553.870138454 0loss/FRt1/assignment_loss‚1.9562149461287217 0loss/FRt2/assignment_loss‚0.9407664134330356 2loss/SRqqt2/assignment_loss‚0.6540241061544899 2loss/FBt2/assignment_loss‚0.013615696973171356 /loss/FBt2/detection_loss‚0.8625436292558506 trainer/global_step‚1749 % _timestamp‚1.7845083802948377e+09 _step‚143 2loss/FBt1/assignment_loss‚0.011306853905308846 /loss/FRt1/detection_loss‚0.5580272990022895 /loss/FRt2/detection_loss‚1.4957047134561023 epoch‚4 2loss/SRqqt1/assignment_loss‚0.6288668590657394 1loss/SRqqt1/detection_loss‚0.8133278270899567 1loss/SRqqt2/detection_loss‚0.7107934865806592» Q¦¼ ˆ 0loss/FRt1/assignment_loss‚1.9562149461287217 0loss/FRt2/assignment_loss‚0.9407664134330356 2loss/SRqqt2/assignment_loss‚0.6540241061544899 2loss/FBt2/assignment_loss‚0.013615696973171356 /loss/FBt2/detection_loss‚0.8625436292558506 trainer/global_step‚1749 % _timestamp‚1.7845083802948377e+09 _step‚143 2loss/FBt1/assignment_loss‚0.011306853905308846 /loss/FRt1/detection_loss‚0.5580272990022895 /loss/FRt2/detection_loss‚1.4957047134561023 epoch‚4 2loss/SRqqt1/assignment_loss‚0.6288668590657394 1loss/SRqqt1/detection_loss‚0.8133278270899567 1loss/SRqqt2/detection_loss‚0.7107934865806592 /loss/FBt1/dete¥/1hction_loss‚0.4861992402260916 %loss/total_loss‚9.131391071271416 _runtime‚1553.870138454óî>U½ ‚(:Ê âßõÒª–ÓË( proc.memory.availableMB‚ 739.26953125* proc.memory.percent‚18.064498901367188 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚4005.8271484375 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚2671.73486328125 disk.rbd0.out‚0 cpu‚25.613223726380173 proc.memory.rssMB‚ 2959.6875 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚ 50.33984375 network.sent‚ 2.623067e+06 network.recv‚438477& disk.nvme0c0n1.out‚10646.935546875 disk.nvme0n1.in‚ 0.00390625# disk.dm-0.out‚11873.89013671875 disk.dm-1.in‚0( disk./.usagePercent‚76.0982548835281% disk./.usageGB‚1168.2984580993652$ disk.nvme0n1.out‚10646.935546875$ disk.nvme1n1.out‚4005.8271484375 disk.rbd0.in‚0# memory_percent‚95.4878568649292éu0㜾 ‚(:‘ âßõÒ¼‹Ì gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.634' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.21133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡¬ÁX¿ ‚(:Í ñßõÒ“€·Ë$ disk.nvme1n1.out‚4006.5341796875" disk.dm-1.out‚2671.79736328125 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625# memory_percent‚95.4878568649292 disk.nvme1n1.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 739.26953125 cpu‚25.611692512549112 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚13747.8271484375 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚12521.36083984375* proc.memory.percent‚18.064498901367188 network.sent‚ 2.673435e+06) disk./.usagePercent‚76.14736676584954 disk.md0.in‚0 disk.md0.out‚ 50.87109375( disk.nvme0c0n1.out‚12521.36083984375 proc.memory.rssMB‚ 2959.6875 proc.cpu.threads‚17 network.recv‚442530$ disk./.usageGB‚1169.052448272705& disk.nvme1c1n1.out‚4006.5341796875};,‹‘À ‚(:† ñßõÒ­‰ùË gpu.0.gpu‚92.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚278.07' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.69 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o:é?RÁ ‚(:Ç €àõÒ…Ù¥Ë! disk.nvme1n1.out‚ 4007.2421875 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚13750.69580078125 disk.rbd0.out‚0 cpu‚25.610342655742325 proc.cpu.threads‚17 network.sent‚ 2.678223e+06 network.recv‚445483) disk./.usagePercent‚76.14740652169142% disk.nvme0n1.out‚12524.3798828125# disk.nvme1c1n1.out‚ 4007.2421875" disk.dm-1.out‚2671.90185546875# memory_percent‚95.4878568649292( proc.memory.availableMB‚ 739.26953125% disk./.usageGB‚1169.0530586242676 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 51.22265625 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚12524.3798828125 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.memory.rssMB‚ 2959.6875* proc.memory.percent‚18.064498901367188ÂÄJªž ‚(:“ €àõÒÉ»ìË gpu.0.gpu‚62.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.237' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.74566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NÜR2à - _wandbruntime‚1595  _runtime‚1595tVøÙÇÄ Á trainer/global_step‚1799 % _timestamp‚1.7845084221403506e+09 _runtime‚1595.716504613 _step‚144 & lr-AdamW/pg1‚0.0006067971101037011 & lr-AdamW/pg2‚0.0006067971101037011W ámÌÅ Æ trainer/global_step‚1799 % _timestamp‚1.7845084221403506e+09 _runtime‚1595.716504613 _step‚144 & lr-AdamW/pg1‚0.0006067971101037011 & lr-AdamW/pg2‚0.0006067971101037011×w2Æ - _wandbruntime‚1595  _runtime‚1595ÁÑ`–ß'Ç ÙO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276781553064960 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7443065 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14522535 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17780810 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚273095788617072 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59943030766080 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7442768 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14522537 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3143277 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274699071448576 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276781553064960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59943049378304 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845084221410673e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32293202 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2082498655744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2094938484736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚275175479667022 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2079691049950 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17780810 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3143280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273095771577712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚145 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2082840079872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274699054409216 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4299491 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2094835312640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2079350488676 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17780810 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚273095788617072 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59943030766080 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7442768 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14522537 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3143277 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274699071448576 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276781553064960 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59943049378304 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845084221410673e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32293202 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2082498655744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2094938484736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚275175479667022‘gZ2É - _wandbruntime‚1596  _runtime‚15969ÙyÊ âM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32320810 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32310651 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 $ _timestamp‚1.784508422751567e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17790480 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17780323 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274847345950720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276931296136704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2083950185984 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3145014 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2095954657280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚59976211036160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚273243202245184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4301732 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚146 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2083608754688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2083950185984 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17780323 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276930937666048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚1596.327374413 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2083608754688 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7447028 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚1799 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚273243219284544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32320810 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14530328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32310651 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14530330 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3145011 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚59976229648384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚275324018722858 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2080799438314 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14530330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2096055725056 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62072285uto373440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274847345950720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚275323661122112 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14530328 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4302014 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7446743 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚276931296136704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274847328911360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62072165693440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0È7‡{í&Ë çM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7446743 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚276931296136704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274847328911360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62072165693440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32320810 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32310651 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 $ _timestamp‚1.784508422751567e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17790480 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17780323 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274847345950720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276931296136704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2083950185984 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3145014 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2095954657280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚59976211036160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚273243202245184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4301732 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚146 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2083608754688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2083950185984 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17780323 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276930937666048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚1596.327374413 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2083608754688 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7447028 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚1799 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚273243219284544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32320810 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14530328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32310651 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14530330 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3145011 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚59976229648384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚275324018722858 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2080799438314 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14530330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2096055725056 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62072285373440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274847345950720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚275323661122112 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14530328 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4302014 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184’”þñß2Ì - _wandbruntime‚1596  _runtime‚1596fãcÄÍ  1loss/SRqqt2/detection_loss‚0.7827211630227939 trainer/global_step‚1799 _runtime‚1596.329627766 /loss/FRt2/detection_loss‚1.5887986586660896 _step‚147 2loss/SRqqt1/assignment_loss‚0.6859788376926172 2loss/SRqqt2/assignment_loss‚0.6590856932641173 /loss/FBt1/detection_loss‚0.5343449984511333 %loss/total_loss‚9.119297924496738 epoch‚4 /loss/FRt1/assignment_loss‚1.979237722161928 .loss/FBt2/detection_loss‚0.587849530416566 % _timestamp‚1.7845084227542574e+09 0loss/FRt2/assignment_loss‚0.9562489889455184 1loss/FBt1/assignment_loss‚0.03202196800917329 3loss/FBt2/assignment_loss‚0.0027682944093168614 /loss/FRt1/detection_loss‚0.5278535633952438 1loss/SRqqt1/detection_loss‚0.7823885060622406 Û]z Î † 2loss/SRqqt1/assignment_loss‚0.6859788376926172 2loss/SRqqt2/assignment_loss‚0.6590856932641173 /loss/FBt1/detection_loss‚0.5343449984511333 %loss/total_loss‚9.119297924496738 epoch‚4 /loss/FRt1/assignment_loss‚1.979237722161928 .loss/FBt2/detection_loss‚0.587849530416566 % _timestamp‚1.7845084227542574e+09 0loss/FRt2/assignment_loss‚0.9562489889455184 1loss/FBt1/assignment_loss‚0.03202196800917329 3loss/FBt2/assignment_loss‚0.0027682944093168614 /loss/FRt1/detection_loss‚0.5278535633952438 1loss/SRqqt1/detection_loss‚0.7823885060622406 1loss/SRqqt2/detection_loss‚0.7827211630227939 trainer/global_step‚1799 _runtime‚1596.329627766 /loss/FRt2/detection_loss‚1.5887986586660896 _step‚147“ûh©ÎRÏ ‚(:Ç àõÒÇé™Ë* proc.memory.percent‚18.064498901367188 cpu‚25.60839960810621 proc.cpu.threads‚17 network.sent‚ 2.681145e+06 disk.md0.in‚0 disk.md0.out‚ 51.94921875# disk.dm-0.out‚14618.21923828125 disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚95.48807144165039( disk./.usagePercent‚76.1288527187589 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚13174.9091796875# disk.nvme1n1.out‚4225.833984375" disk.dm-1.out‚2671.96435546875 disk.rbd0.in‚0& proc.memory.availableMB‚ 739.234375 proc.memory.rssMB‚ 2959.6875$ disk./.usageGB‚1168.768211364746% disk.nvme1c1n1.out‚4225.833984375 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚13174.9091796875 network.recv‚448795 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.003906256 C;‘Ð ‚(:† àõÒËöÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.429' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.143 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03´“Ñ ‚(:ˆ žàõÒ²ÎË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚255.426' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚85.142 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QuÞ/WÒ ‚(:Ì žàõÒñ¥çË# disk.dm-0.out‚14625.11083984375 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚18.064498901367188 disk.md0.out‚ 52.41015625 disk.dm-1.in‚0 disk.dm-1.out‚2672.052734375 proc.memory.rssMB‚ 2959.6875 cpu‚25.60723317223067 disk.nvme1n1.in‚0% disk.nvme1n1.out‚4227.24560546875$ disk./.usageGB‚1168.768154144287' disk.nvme0c0n1.out‚13181.4677734375 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚13181.4677734375 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625 network.sent‚ 2.745405e+06) disk./.usagePercent‚76.12884899164874$ memory_percent‚95.48962116241455( proc.memory.availableMB‚ 738.98046875 proc.cpu.threads‚17 disk.md0.in‚0 network.recv‚452672 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚4227.24560546875IÁKéTÓ ‚(:É ­àõÒÚÆÆË network.recv‚455676% disk./.usageGB‚1168.7687759399414 disk.nvme1n1.in‚0% disk.nvme1n1.out‚4227.95556640625 disk.md0.out‚ 52.91796875 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.dm-1.out‚2672.091796875# disk.dm-0.out‚14626.03271484375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚4227.95556640625$ memory_percent‚95.48959732055664& proc.memory.availableMB‚ 738.984375 network.sent‚ 2.75002e+06) disk./.usagePercent‚76.12888949291265 disk.md0.in‚0& disk.nvme0c0n1.out‚13182.794921875$ disk.nvme0n1.out‚13182.794921875 disk.nvme1c1n1.in‚0 proc.memory.rssMB‚ 2959.6875* proc.memory.percent‚18.064498901367188 cpu‚25.606570159235346 proc.cpu.threads‚17¥xþ`‘Ô ‚(:† ­àõÒÿêÂË gpu.0.gpu‚3.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚288.954' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.318 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õV—2Õ - _wandbruntime‚1638  _runtime‚1638TþËÇÖ Á % _timestamp‚1.7845084644879336e+09 _runtime‚1638.063469279 _step‚148 & lr-AdamW/pg1‚0.0006062399584582438 & lr-AdamW/pg2‚0.0006062399584582438 trainer/global_step‚1849êšohÌ× Æ & lr-AdamW/pg1‚0.0006062399584582438 & lr-AdamW/pg2‚0.0006062399584582438 trainer/global_step‚1849 % _timestamp‚1.7845084644879336e+09 _runtime‚1638.063469279 _step‚148”žé2Ø - _wandbruntime‚1638  _runtime‚1638ëb˜¯ß'Ù ÙO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚284251205737472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61602062882304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282602138653396 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14912187 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18265462 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2150819675136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280467321990672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63752760773120 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _step‚149 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33177649 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14912187 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2134833702084 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚284251205737472 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284250847273984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2138409181696 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4414853 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2135174263358 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2138067757568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18265462 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3229980 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0åÉÂ`Ú ÞO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚282112779516416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282112796555776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2138067757568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2150716503040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚284251205737472 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284250847273984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2138409181696 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4414853 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2135174263358 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2138067757568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18265462 WKDeviceStatsMonÌ×Y„itor.on_train_batch_start/inactive_split.large_pool.allocated‚3229980 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚284251205737472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61602062882304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282602138653396 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14912187 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18265462 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2150819675136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚280467321990672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63752760773120 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _step‚149 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33177649 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14912187 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2134833702084 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33195112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14919978 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4417153 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2139519287808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14919978 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282261054018560 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚284400231875072 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282261071057920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2151938277376 _step‚150 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2135942090336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2139177856512 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7648595 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7845084651126747e+09 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7648867 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3231714 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63787057555456 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2139519287808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61635224540160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33184953 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282261071057920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33195112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14919980 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33184953 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18275132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2151833015296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282751035309866 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280614735618784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚284400590345728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61635243152384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚1849 _runtime‚1638.688429728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7648595 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7845084651126747e+09 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7648867 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3231714 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63787057555456 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2139519287808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61635224540160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33184953 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282261071057920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33195112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14919980 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33184953 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18275132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2151833015296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282751035309866 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280614735618784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚284400590345728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61635243152384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚1849 _runtime‚1638.688429728 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33195112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14919978 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4417153 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2139519287808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14919978 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282261054018560 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚284400231875072 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282261071057920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2151938277376 _step‚150 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2135942090336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0–vï(|2Þ - _wandbruntime‚1638  _runtime‚1638êŠOóß ‚ 0loss/FRt2/assignment_loss‚0.9730135795496238 2loss/SRqqt1/assignment_loss‚0.6054704217373638 1loss/FBt1/assignment_loss‚0.03546105739185417 /loss/FRt2/detection_loss‚1.6798439510397742 % _timestamp‚1.7845084651137302e+09 _step‚151 1loss/SRqqt1/detection_loss‚0.7588718051879059 trainer/global_step‚1849 0loss/FRt1/assignment_loss‚1.9547158373334936 2loss/SRqqt2/assignment_loss‚0.6497867395345172 /loss/FRt1/detection_loss‚0.5112355661119063 1loss/SRqqt2/detection_loss‚0.7672705284609262 /loss/FBt1/detection_loss‚0.6123438346284357 %loss/total_loss‚9.305329869699294 _runtime‚1638.689020739 2loss/FBt2/assignment_loss‚0.013783920823262885 /loss/FBt2/detection_loss‚0.7435326279002301 epoch‚4c÷ à ‡ 0loss/FRt1/assignment_loss‚1.9547158373334936 2loss/SRqqt2/assignment_loss‚0.6497867395345172 /loss/FRt1/detection_loss‚0.5112355661119063 1loss/SRqqt2/detection_loss‚0.7672705284609262 /loss/FBt1/detection_loss‚0.6123438346284357 %loss/total_loss‚9.305329869699294 _runtime‚1638.689020739 2loss/FBt2/assignment_loss‚0.013783920823262885 /loss/FBt2/detection_loss‚0.7435326279002301 epoch‚4 0loss/FRt2/assignment_loss‚0.9730135795496238 2loss/SRqqt1/assignment_loss‚0.6054704217373638 1loss/FBt1/assignment_loss‚0.03546105739185417 /loss/FRt2/detection_loss‚1.6798439510397742 % _timestamp‚1.7845084651137302e+09 _step‚151 1loss/SRqqt1/detection_loss‚0.7588718051879059 trainer/global_step‚1849—Û6çgQá ‚(:Æ ¼àõÒæÅÌË) disk./.usagePercent‚76.12889123223073% disk./.usageGB‚1168.7688026428223( disk.nvme0c0n1.out‚13191.52001953125 disk.md0.in‚0 proc.cpu.threads‚17 network.recv‚458698" disk.dm-0.out‚14636.4306640625 disk.dm-1.in‚0" disk.dm-1.out‚2672.18408203125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0" disk.nvme1c1n1.out‚ 4230.640625 disk.dm-0.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 4230.640625 disk.md0.out‚ 53.34765625$ memory_percent‚95.48830986022949' proc.memory.availableMB‚ 739.1953125 cpu‚25.607109651453825& disk.nvme0n1.out‚13191.52001953125 network.sent‚ 2.765039e+06 disk.rbd0.out‚0 proc.memory.rssMB‚ 2959.6875* proc.memory.percent‚18.064498901367188F­É~žâ ‚(:“ ¼àõÒ¤ä’Ì gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚262.397' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.46566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ¬MzYã ‚(:Î ËàõÒÓÇ’Ë network.recv‚463304 disk.md0.out‚ 53.6171875 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚13277.64794921875 proc.memory.rssMB‚ 2959.6875* proc.memory.percent‚18.064498901367188 proc.cpu.threads‚17) disk./.usagePercent‚76.14451378724645% disk./.usageGB‚1169.0086479187012 disk.nvme1n1.in‚0" disk.dm-0.out‚14756.5361328125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚13277.64794921875$ memory_percent‚95.48842906951904 network.sent‚ 2.814556e+06 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚2672.22705078125 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚4265.2451171875( proc.memory.availableMB‚ 739.17578125$ disk.nvme1n1.out‚4265.2451171875 disk.dm-1.in‚0 cpu‚25.605722413306296Ooဓä ‚(:ˆ ËàõÒ™òëË gpu.0.gpu‚80.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚264.669' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.223 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0͈PYå ‚(:Î ÚàõÒ þ–Ë! disk.dm-1.out‚2672.3544921875' disk.nvme0c0n1.out‚13360.4794921875 disk.nvme1n1.in‚0 disk.md0.out‚ 54.11328125$ memory_percent‚95.48847675323486 proc.memory.rssMB‚ 2959.6875 cpu‚25.60419682399307% disk./.usageGB‚1169.0070838928223 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme0n1.out‚13360.4794921875 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625( proc.memory.availableMB‚ 739.16796875* proc.memory.percent‚18.064498901367188 proc.cpu.threads‚17 network.recv‚466454) disk./.usagePercent‚76.14441191290163 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚4319.95458984375 network.sent‚ 2.820424e+06 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme1c1n1.out‚4319.95458984375# disk.dm-0.out‚14892.87158203125™îêKžæ ‚(:“ ÚàõÒ®ÕåË gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚262.263' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.42099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cÃ>2ç - _wandbruntime‚1680  _runtime‚1680(Çè Á _step‚152 & lr-AdamW/pg1‚0.0006056636029067566 & lr-AdamW/pg2‚0.0006056636029067566 trainer/global_step‚1899 % _timestamp‚1.7845085070264642e+09 _runtime‚1680.604029239ÀߊÌé Æ trainer/global_step‚1899 % _timestamp‚1.7845085070264642e+09 _runtime‚1680.604029239 _step‚152 & lr-AdamW/pg1‚0.0006056636029067566 & lr-AdamW/pg2‚0.0006056636029067566˜á7rp2ê - _wandbruntime‚1680  _runtime‚1680< +„åë ÙO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15301835 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2190657476766 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚1899 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290029512841038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚291720141483008 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2206558234624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceSNdêãú#tatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289526521662976 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚291720499946496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _runtime‚1680.604124109 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18750114 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3316677 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚287838855364272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15301837 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2190316915492 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287838838324912 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7846017 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65467739890176 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65467616008704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290029155240404 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18739971 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2193978283520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63261076386304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚291720141483008 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34051951 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18750114 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289526504623616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63261057774080 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4529340 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2206663503872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2193636859392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15301837 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7846306 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34041806 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚291720499946496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4529626 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2193636859392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845085070270722e+09 _step‚153 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚291720499946496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _runtime‚1680.604124109 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18750114 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3316677 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚287838855364272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15301837 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2190316915492 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287838838324912 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7846017 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65467739890176 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65467616008704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290029155240404 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18739971 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2193978283520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63261076386304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚291720141483008 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34051951 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18750114 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289526504623616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63261057774080 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4529340 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2206663503872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2193636859392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15301837 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7846306 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34041806 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚291720499946496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4529626 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2193636859392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845085070270722e+09 _step‚153 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚291720141483008 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2206558234624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3™Êýxx2í - _wandbruntime‚1681  _runtime‚1681øŸè&î âM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18749627 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18759784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2191425303744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289674796165120 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34069414 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34059255 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65502034347008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2207674525696 % _timestamp‚1.7845085076366158e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34069414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18759784 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3318411 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2207777690624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 _step‚154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290178051896874 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4531640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63294238044160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15309630 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2194746958336 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚291869884554752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚291869526084096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63294256656384 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34059255 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15309628 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289674779125760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4531927 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2195088389632 _runtime‚ 1681.21232936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2195088389632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚287986268992384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15309628 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7850051 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2191765865130 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290177694296128 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚291869884554752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15309630 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18749627 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3318414 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65501912569856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289674796165120ù—èªå ï çM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289674796165120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18749627 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18759784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2191425303744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289674796165120 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34069414 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34059255 [HDeviceStat²ßü€sMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65502034347008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2207674525696 % _timestamp‚1.7845085076366158e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34069414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18759784 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3318411 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2207777690624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 _step‚154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290178051896874 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4531640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63294238044160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15309630 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2194746958336 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚291869884554752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚291869526084096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63294256656384 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34059255 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15309628 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289674779125760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4531927 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2195088389632 _runtime‚ 1681.21232936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2195088389632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚287986268992384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15309628 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7850051 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2191765865130 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290177694296128 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚291869884554752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15309630 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18749627 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3318414 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65501912569856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452š5Õ‹ž2ð - _wandbruntime‚1681  _runtime‚1681ô¢Zñ ÿ trainer/global_step‚1899 2loss/SRqqt2/assignment_loss‚0.6075876212739323 3loss/FBt2/assignment_loss‚0.0007272502811972388 0loss/FBt1/assignment_loss‚0.0226669663729593 /loss/FRt2/detection_loss‚1.6004065316742038 1loss/SRqqt1/detection_loss‚0.8368208846260211 .loss/FBt1/detection_loss‚0.576405050627289 % _timestamp‚1.7845085076395876e+09 _runtime‚1681.214909124 0loss/FRt1/assignment_loss‚1.9037393439634174 0loss/FRt2/assignment_loss‚1.0456911996966267 epoch‚4 _step‚155 2loss/SRqqt1/assignment_loss‚0.6495121037550594 .loss/FRt1/detection_loss‚0.533424901282021 0loss/SRqqt2/detection_loss‚0.735851111784884 /loss/FBt2/detection_loss‚0.6778334369631233 %loss/total_loss‚9.190666402300735ë“bm ò „ 0loss/FRt2/assignment_loss‚1.0456911996966267 epoch‚4 _step‚155 2loss/SRqqt1/assignment_loss‚0.6495121037550594 .loss/FRt1/detection_loss‚0.533424901282021 0loss/SRqqt2/detection_loss‚0.735851111784884 /loss/FBt2/detection_loss‚0.6778334369631233 %loss/total_loss‚9.190666402300735 trainer/global_step‚1899 2loss/SRqqt2/assignment_loss‚0.6075876212739323 3loss/FBt2/assignment_loss‚0.0007272502811972388 0loss/FBt1/assignment_loss‚0.0226669663729593 /loss/FRt2/detection_loss‚1.6004065316742038 1loss/SRqqt1/detection_loss‚0.8368208846260211 .loss/FBt1/detection_loss‚0.576405050627289 % _timestamp‚1.7845085076395876e+09 _runtime‚1681.214909124 0loss/FRt1/assignment_loss‚1.9037393439634174›ËNSTó ‚(:É éàõÒøºðË disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625% disk.nvme1n1.out‚4320.54736328125) disk./.usagePercent‚76.14441067053157 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0! disk.dm-1.out‚2672.4677734375 disk.nvme1n1.in‚0 proc.memory.rssMB‚ 2959.6875* proc.memory.percent‚18.064498901367188$ disk./.usageGB‚1169.007064819336# disk.dm-0.out‚14893.72314453125& disk.nvme0c0n1.out‚13361.599609375$ disk.nvme0n1.out‚13361.599609375' disk.nvme1c1n1.out‚4320.54736328125' proc.memory.availableMB‚ 739.0859375 cpu‚25.602823283550066 proc.cpu.threads‚17 network.recv‚470197 disk.md0.out‚ 54.453125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0$ memory_percent‚95.48897743225098 network.sent‚ 2.837023e+06_þ³žô ‚(:“ éàõÒ‚©°Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚289.469' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.48966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è»-Lõ ‚(:Á øàõÒ†ÚÖË% disk./.usageGB‚1169.0071334838867 disk.dm-1.in‚0 disk.dm-1.out‚ 2672.54296875 disk.nvme0n1.in‚ 0.00390625 disk.nvme0n1.out‚ 13364.46875 disk.nvme1n1.in‚0% disk.nvme1n1.out‚4321.32275390625 disk.md0.out‚ 54.8046875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚14896.54052734375* proc.memory.percent‚18.064498901367188 cpu‚25.60121191321082" disk.nvme0c0n1.out‚ 13364.46875$ memory_percent‚95.48914432525635 proc.cpu.threads‚17 network.sent‚ 2.888319e+06 network.recv‚474017) disk./.usagePercent‚76.14441514306378 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚4321.32275390625( proc.memory.availableMB‚ 739.05859375 proc.memory.rssMB‚ 2959.6875áZäÄžö ‚(:“ øàõÒæ Ì gpu.0.gpu‚92.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚282.025' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.00833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈærŸ2÷ - _wandbruntime‚1722  _runtime‚1722ÈE=ÎÆø À & lr-AdamW/pg1‚0.0006050680804674794 & lr-AdamW/pg2‚0.0006050680804674794 trainer/global_step‚1949 % _timestamp‚1.7845085490410807e+09 _runtime‚ 1722.61813135 _step‚156)Q‡jËù Å & lr-AdamW/pg2‚0.0006050680804674794 trainer/global_step‚1949 % _timestamp‚1.7845085490410807e+09 _runtime‚ 1722.61813135 _step‚156 & lr-AdamW/pg1‚0.0006050680804674794œJ>á2ú - _wandbruntime‚1722  _runtime‚1722Dº*¿Þ'û ØO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296940229730816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2262441404416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2245800128900 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2249547385344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2249205961216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚157 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296940246770176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299189435692032 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67182512682496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2246140690174 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34916108 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4644110 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19224623 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34926253 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚297456529428046 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.784508549043148e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚299189435692032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296940246770176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19224623 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4643826 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚1722.618673381 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2249205961216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64920071278080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295210371698512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15691487 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15691485 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚295210388737872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296940229730816 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8047203 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299189794155520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8047490 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19234766 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528NñIýã'ü ÝO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2246140690174 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34916108 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4644110 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19224623 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34926253 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚297456529428046 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.784508549043148e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚299189435692032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296940246770176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19224623 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4643826 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚1722.618673381 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2249205961216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64920071278080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295210371698512 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15691487 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15691485 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚295210388737872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296940229730816 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8047203 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299189794155520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8047490 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19234766 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296940229730816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2262441404416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2245800128900 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2249547385344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2249205961216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚157 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296940246770176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299189435692032 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67182512682496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2aØ4é2ý - _wandbruntime‚1723  _runtime‚1723À·Ôƒþ âM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3405111 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚299339178763776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2250657491456 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67216933962240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2250316060160 WDDeviceStatsMonitor.on_train_batch_end/inactive_vÉ5)e#split_bytes.all.freed‚67216807990784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚295357802365984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚158 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34933557 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15699280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34943716 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚297088521272320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2250316060160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19244436 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚295357819405344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8051491 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19244436 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4646093 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚1949 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15699278 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2247249078538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3405114 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19234279 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚297088521272320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15699278 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2263556442624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8051204 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64953251548160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2250657491456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34943716 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚297605068483882 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296í|Bí&ÿ çM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8051204 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64953251548160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2250657491456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34943716 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚297605068483882 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3405111 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚299339178763776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2250657491456 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67216933962240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2250316060160 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67216807990784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚295357802365984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚158 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34933557 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15699280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34943716 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚297088521272320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2250316060160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19244436 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚295357819405344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8051491 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19244436 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4646093 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚1949 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15699278 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2247249078538 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3405114 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19234279 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚297088521272320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15699278 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2263556442624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0žvw¬2€ - _wandbruntime‚1723  _runtime‚1723k-º  „ /loss/FRt1/detection_loss‚0.5276107979934663 $ _timestamp‚1.784508549961162e+09 0loss/FRt1/assignment_loss‚2.0143688709938536 /loss/FRt2/detection_loss‚1.5917317979492593 1loss/SRqqt2/detection_loss‚0.7434131485878552 epoch‚4 0loss/FRt2/assignment_loss‚1.0392214194286702 2loss/SRqqt2/assignment_loss‚0.6432475147417244 2loss/FBt1/assignment_loss‚0.014138849628549503 3loss/FBt2/assignment_loss‚0.0016822962237147915 1loss/SRqqt1/detection_loss‚0.7605951781205449 0loss/FBt1/detection_loss‚0.48200774340269464 /loss/FBt2/detection_loss‚0.7823957842351361 2loss/SRqqt1/assignment_loss‚0.5772248453960368 %loss/total_loss‚9.177638246701507 trainer/global_step‚1949 _runtime‚1723.536467273 _step‚159«‡ÈÅ‚ ‰ epoch‚4 0loss/FRt2/assignment_loss‚1.0392214194286702 2loss/SRqqt2/assignment_loss‚0.6432475147417244 2loss/FBt1/assignment_loss‚0.014138849628549503 3loss/FBt2/assignment_loss‚0.0016822962237147915 1loss/SRqqt1/detection_loss‚0.7605951781205449 0loss/FBt1/detection_loss‚0.48200774340269464 /loss/FBt2/detection_loss‚0.7823957842351361 2loss/SRqqt1/assignment_loss‚0.5772248453960368 %loss/total_loss‚9.177638246701507 trainer/global_step‚1949 _runtime‚1723.536467273 _step‚159 /loss/FRt1/detection_loss‚0.5276107979934663 $ _timestamp‚1.784508549961162e+09 0loss/FRt1/assignment_loss‚2.0143688709938536 /loss/FRt2/detection_loss‚1.5917317979492593 1loss/SRqqt2/detection_loss‚0.7434131485878552ŸÍ²Vƒ ‚(:Ë ‡áõÒ›äéË* proc.memory.percent‚18.064498901367188 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚4321.96533203125 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.out‚ 2672.61328125( proc.memory.availableMB‚ 492.74609375) disk./.usagePercent‚76.14446061380794 disk.rbd0.out‚0& disk.nvme0c0n1.out‚13365.748046875# disk.dm-0.out‚14897.44287109375 disk.dm-1.in‚0 cpu‚25.600048206425935 disk.rbd0.in‚0' disk.nvme1c1n1.out‚4321.96533203125$ memory_percent‚96.99251651763916 proc.cpu.threads‚17 network.sent‚ 2.904185e+06 network.recv‚477660% disk./.usageGB‚1169.0078315734863 disk.nvme0n1.in‚ 0.00390625$ disk.nvme0n1.out‚13365.748046875 disk.md0.out‚ 55.23828125 proc.memory.rssMB‚ 2959.6875 :Ôpž„ ‚(:“ ‡áõÒºù­Ì gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚272.534' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.84466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09 §ZU… ‚(:Ê –áõÒÓƳË( proc.memory.availableMB‚ 492.72265625 proc.memory.rssMB‚ 2959.6875* proc.memory.percent‚18.064498901367188 network.recv‚481442) disk./.usagePercent‚76.14446334702207 disk.rbd0.in‚0% disk.nvme0c0n1.out‚13367.27734375 cpu‚25.598494905696672 network.sent‚ 2.955515e+06 disk.md0.out‚ 55.6640625 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625# disk.nvme0n1.out‚13367.27734375' disk.nvme1c1n1.out‚4322.68212890625$ memory_percent‚96.99265956878662 proc.cpu.threads‚17 disk.md0.in‚0! disk.dm-1.out‚2672.7080078125 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk./.usageGB‚1169.0078735351562# disk.dm-0.out‚14898.69384765625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme1n1.out‚4322.68212890625¾\ ž† ‚(:“ –áõÒ× üË gpu.0.gpu‚75.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚252.737' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.24566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§µtXX‡ ‚(:Í ¥áõÒò¬ÌË disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚4323.36376953125 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 492.69140625 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0# memory_percent‚96.9928503036499 proc.memory.rssMB‚ 2959.6875 proc.cpu.threads‚17) disk./.usagePercent‚76.14446384397009 disk.md0.out‚ 56.1328125! disk.dm-1.out‚2672.7939453125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚13368.43505859375% disk.nvme1n1.out‚4323.36376953125* proc.memory.percent‚18.064498901367188 cpu‚25.597231543420477 network.sent‚ 2.960415e+06% disk./.usageGB‚1169.0078811645508 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 network.recv‚485262! disk.dm-0.out‚14899.451171875& disk.nvme0n1.out‚13368.43505859375?Èüxˆ ‚(:’ ¥áõÒ¬¨¢Ì gpu.0.gpu‚80.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.26' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.41999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô°}‹2‰ - _wandbruntime‚1764  _runtime‚1764d‘ßNÇŠ Á & lr-AdamW/pg1‚0.0006044534293897058 & lr-AdamW/pg2‚0.0006044534293897058 trainer/global_step‚1999 % _timestamp‚1.7845085913794405e+09 _runtime‚1764.955035673 _step‚160!0ß:Ì‹ Æ & lr-AdamW/pg1‚0.0006044534293897058 & lr-AdamW/pg2‚0.0006044534293897058 trainer/global_step‚1999 % _timestamp‚1.7845085913794405e+09 _runtime‚1764.955035673 _step‚160 h)ÖT2Œ - _wandbruntime‚1764  _runtime‚1764|Õ†e  ÙO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16081135 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304353954838016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35790410 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8248459 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4758658 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845085913803654e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19719418 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19719418 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2301623903582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302581905072112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35800555 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2318371585024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚304353971877376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚161 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚304353971877376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306659088364544 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306658729901056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2304775063040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3490080 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19709275 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19709275 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68897474979328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304883546015054 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZHDeviceStÀóz¿ atsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2301283342308 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8248738 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66579084782080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304353954838016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35790410 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8248459 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4758658 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845085913803654e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19719418 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19719418 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2301623903582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302581905072112 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35800555 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2318371585024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚304353971877376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚161 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚304353971877376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306659088364544 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306658729901056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2304775063040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3490080 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19709275 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19709275 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68897474979328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304883546015054 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2301283342308 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8248738 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66579084782080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8252420 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚1999 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚305031727470144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2302732291946 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚162 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚304502246379520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2319490678784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2306226593280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302729352778944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16088930 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8252721 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66612265052160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16088928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306808114502144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4760609 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306808472972800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35807859 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16088928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305032085070890 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306808472972800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35807859 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19729088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302729335739584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35818018 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19718931 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656'}¸î&‘ èM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2306226593280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302729352778944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16088930 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8252721 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66612265052160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16088928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306808114502144 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4760609 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306808472972800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35807859 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16088928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305032085070890 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306808472972800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35807859 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19729088 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302729335739584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35818018 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19718931 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68931774343168 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68931646274560 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4760907 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8252420 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚1999 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚305031727470144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2302732291946 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚162 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚304502246379520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2319490678784¢¦è¦“*’ - _wandbruntime‚1765  _runtimD\§´e‚1765jk¨t“ ÿ trainer/global_step‚1999 _runtime‚1765.567946392 0loss/FRt1/assignment_loss‚1.9725619120929474 1loss/FBt1/assignment_loss‚0.03663027057526832 0loss/SRqqt1/detection_loss‚0.782752661154718 1loss/SRqqt2/detection_loss‚0.7377214785725847 /loss/FBt2/detection_loss‚0.8032303287647664 %loss/total_loss‚9.310340366158027 0loss/FRt2/assignment_loss‚0.9617362253788941 1loss/SRqqt2/assignment_loss‚0.583785778802405 2loss/FBt2/assignment_loss‚0.001068029949115612 .loss/FRt2/detection_loss‚1.646985145535146 % _timestamp‚1.7845085919926312e+09 _step‚163 2loss/SRqqt1/assignment_loss‚0.6845004991528285 /loss/FRt1/detection_loss‚0.5124062194580138 /loss/FBt1/detection_loss‚0.5869618167213392 epoch‚4®êq ” „ 0loss/SRqqt1/detection_loss‚0.782752661154718 1loss/SRqqt2/detection_loss‚0.7377214785725847 /loss/FBt2/detection_loss‚0.8032303287647664 %loss/total_loss‚9.310340366158027 0loss/FRt2/assignment_loss‚0.9617362253788941 1loss/SRqqt2/assignment_loss‚0.583785778802405 2loss/FBt2/assignment_loss‚0.001068029949115612 .loss/FRt2/detection_loss‚1.646985145535146 % _timestamp‚1.7845085919926312e+09 _step‚163 2loss/SRqqt1/assignment_loss‚0.6845004991528285 /loss/FRt1/detection_loss‚0.5124062194580138 /loss/FBt1/detection_loss‚0.5869618167213392 epoch‚4 trainer/global_step‚1999 _runtime‚1765.567946392 0loss/FRt1/assignment_loss‚1.9725619120929474 1loss/FBt1/assignment_loss‚0.03663027057526832£ËïrÞ2• ‚(:§ ´áõÒê¾ÁË disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.md0.out‚ 56.5703125$ memory_percent‚97.00148105621338) proc.memory.percent‚18.07255744934082 disk.dm-1.in‚0 disk.nvme0c0n1.out‚13371 disk.nvme1n1.out‚4324.25 disk.md0.in‚0# disk.dm-0.out‚14901.81103515625( proc.memory.availableMB‚ 491.27734375" proc.memory.rssMB‚ 2961.0078125 proc.cpu.threads‚17) disk./.usagePercent‚76.14443700877682 disk.nvme1n1.in‚0 cpu‚25.601456077530365 network.sent‚ 2.97618e+06 network.recv‚488457$ disk./.usageGB‚1169.007469177246" disk.dm-1.out‚2672.94384765625 disk.rbd0.in‚0 disk.nvme0n1.out‚13371 disk.nvme1c1n1.out‚4324.25 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625¸ªöÔš– ‚(: ´áõÒÙ¼þË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.0' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.33333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñê»12— - _wandbruntime‚1781  _runtime‚1781{-80ÿ=˜ ù{ @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1535090208053589 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14903882145881653 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5865415930747986 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.663336992263794 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0768764317035675 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 *jet/accuracy_2_of_4‚0.7712300419807434 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5572992563247681 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1733056902885437 jet/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6283615827560425 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8872174620628357 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6089272499084473 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976937174797058 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 +jet/accuracy_3_of_3‚0.07114022970199585 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18046733736991882 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5841241478919983 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08988555520772934 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5895182490348816 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8752455711364746 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7280840873718262 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965855479240417 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9940970540046692 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03719371184706688 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6665281057357788 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12163212150335312 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47362810373306274 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8974300622940063 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960317015647888 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8199357390403748 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5802773237228394 $ _timestamp‚1.784508608226826e+09 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.949853777885437 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 jet/accuracy_3_of_5‚1 _runtime‚1781.802913631 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6272425055503845 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.989047110080719 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6108605265617371 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14506562054157257 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 epoch‚4 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 particle/accuracy_2_of_2‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9001792669296265 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8991857171058655 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6133662462234497 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020578542724251747 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16343185305595398 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8974300622940063 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958359599113464 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3965723514556885 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0768764317035675 *jet/accuracy_1_of_1‚0.2834826409816742 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4922858774662018 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08948558568954468 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6585136651992798 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6453343629837036 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1592686027288437 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03888193890452385 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5746520757675171 )loss/val_total_loss‚8.848293373647323 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17611192166805267 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1550399512052536 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5750462412834167 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6848118901252747 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959182739257812 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7048845887184143 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 /particle/accuracy_4_of_4‚0.9346253275871277 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5729975700378418 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416667819023132 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1808534860610962 6validation_average_jet_accuracy‚0.3311859667301178 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5586226582527161 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_4‚0.9761239886283875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6097466349601746 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9892652630805969 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 particle/accuracy_2_of_5‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.271847665309906 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16927525401115417 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6393948793411255 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49718043208122253 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9952999949455261 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02748001553118229 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_1_of_2‚1 (particle/accuracy‚0.8455907702445984 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5765031576156616 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7092849016189575 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7277264595031738 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16609305143356323 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.679924488067627 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.40831929445266724 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02412358857691288 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.808600127696991 )particle/sensitivity‚0.61140376329422 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.996985137462616 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 +particle/specificity‚0.9096327424049377 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08486271649599075 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4918344020843506 particle/accuracy_3_of_3‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19476725161075592 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1592686027288437 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 )jet/accuracy_1_of_2‚0.739080011844635 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_5_of_5‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.624891459941864 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 particle/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3404127061367035 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2012026458978653 &particle/f_score‚0.629688024520874 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16122931241989136 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23033449053764343 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13758178055286407 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974562525749207 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18703880906105042 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5706967115402222 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6449008584022522 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6627510786056519 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930664896965027 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1512979418039322 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6292064785957336 .particle/accuracy_3_of_4‚0.997619092464447 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041865348816 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23211583495140076 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33861279487609863 particle/accuracy_1_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 particle/accuracy_3_of_5‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5395833253860474 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1771104633808136 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6683964133262634 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.808600127696991 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980687499046326 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08494964987039566 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995028555393219 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8949923515319824 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17360880970954895 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21434229612350464 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +jet/accuracy_3_of_4‚0.09385798871517181 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3884086012840271 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_2_of_3‚0.5295385122299194 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7279993295669556 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8852229118347168 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6555547118186951 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 _step‚164 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5765031576156616 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.25269103050231934 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012922720983624458 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6337713599205017 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1337585598230362 +jet/accuracy_2_of_2‚0.13995811343193054 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8450931310653687 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6548279523849487 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12821058928966522 trainer/global_step‚1999c§3®´6™ þ{ 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995028555393219 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8949923515319824 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17360880970954895 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21434229612350464 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +jet/accuracy_3_of_4‚0.09385798871517181 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3884086012840271 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_2_of_3‚0.5295385122299194 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7279993295669556 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8852229118347168 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6555547118186951 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 _step‚164 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5765031576156616 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.25269103050231934 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012922720983624458 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6337713599205017 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1337585598230362 +jet/accuracy_2_of_2‚0.13995811343193054 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8450931310653687 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6548279523849487 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12821058928966522 trainer/global_step‚1999 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1535090208053589 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14903882145881653 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5865415930747986 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.663336992263794 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0768764317035675 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 *jet/accuracy_2_of_4‚0.7712300419807434 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5572992563247681 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1733056902885437 jet/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6283615827560425 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8872174620628357 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6089272499084473 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976937174797058 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 +jet/accuracy_3_of_3‚0.07114022970199585 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18046733736991882 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5841241478919983 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08988555520772934 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5895182490348816 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8752455711364746 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7280840873718262 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965855479240417 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9940970540046692 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03719371184706688 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6665281057357788 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12163212150335312 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.47362810373306274 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8974300622940063 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960317015647888 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8199357390403748 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5802773237228394 $ _timestamp‚1.784508608226826e+09 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.949853777885437 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 jet/accuracy_3_of_5‚1 _runtime‚1781.802913631 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6272425055503845 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.989047110080719 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6108605265617371 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14506562054157257 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 epoch‚4 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 particle/accuracy_2_of_2‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9001792669296265 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8991857171058655 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6133662462234497 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020578542724251747 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16343185305595398 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8974300622940063 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958359599113464 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3965723514556885 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0768764317035675 *jet/accuracy_1_of_1‚0.2834826409816742 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4922858774662018 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08948558568954468 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6585136651992798 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6453343629837036 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1592686027288437 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03888193890452385 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5746520757675171 )loss/val_total_loss‚8.848293373647323 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17611192166805267 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1550399512052536 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5750462412834167 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6848118901252747 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959182739257812 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7048845887184143 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 /particle/accuracy_4_of_4‚0.9346253275871277 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5729975700378418 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416667819023132 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1808534860610962 6validation_average_jet_accuracy‚0.3311859667301178 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5586226582527161 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_1_of_4‚0.9761239886283875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6097466349601746 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9892652630805969 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 particle/accuracy_2_of_5‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.271847665309906 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16927525401115417 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6393948793411255 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49718043208122253 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9952999949455261 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02748001553118229 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_1_of_2‚1 (particle/accuracy‚0.8455907702445984 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5765031576156616 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7092849016189575 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7277264595031738 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16609305143356323 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.679924488067627 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.40831929445266724 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02412358857691288 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.808600127696991 )particle/sensitivity‚0.61140376329422 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.996985137462616 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 +particle/specificity‚0.9096327424049377 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08486271649599075 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4918344020843506 particle/accuracy_3_of_3‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19476725161075592 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1592686027288437 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 )jet/accuracy_1_of_2‚0.739080011844635 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_5_of_5‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.624891459941864 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 particle/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3404127061367035 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.453125 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2012026458978653 &particle/f_score‚0.629688024520874 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16122931241989136 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23033449053764343 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13758178055286407 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974562525749207 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18703880906105042 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 9"Purity/0FWÄ!¸PRt1SRqqt*FBt/event_purity‚0.5706967115402222 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6449008584022522 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6627510786056519 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930664896965027 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1512979418039322 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6292064785957336 .particle/accuracy_3_of_4‚0.997619092464447 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041865348816 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23211583495140076 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33861279487609863 particle/accuracy_1_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 particle/accuracy_3_of_5‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5395833253860474 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1771104633808136 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6683964133262634 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.808600127696991 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980687499046326 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08494964987039566 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0¤­¡°Xš ‚(:Í ÃáõÒÒ´ÚË% disk./.usageGB‚1169.0075378417969 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚4570.7529296875 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚2918.42041015625% disk.nvme0n1.out‚13372.4560546875 proc.memory.rssMB‚ 2973.9375 cpu‚25.618232383508737 disk.md0.out‚ 57.4609375# disk.dm-0.out‚14902.41259765625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚13372.4560546875 disk.nvme1c1n1.in‚0 proc.cpu.threads‚17 network.sent‚ 3.038485e+06 network.recv‚492876 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625& disk.nvme1c1n1.out‚4570.7529296875) disk./.usagePercent‚76.14444148130903$ memory_percent‚95.47803401947021( proc.memory.availableMB‚ 740.87890625* proc.memory.percent‚18.151473999023438ÃßDüœ› ‚(:‘ ÃáõÒòá›Ì gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚267.8' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.26666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç±±Þžœ ‚(:“ ÒáõÒΪ¼Ë gpu.0.gpu‚47.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.854' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.95133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ùµ°ö[ ‚(:Ð ÒáõÒ™Ú”Ì! disk.dm-1.out‚2918.5712890625( disk.nvme0c0n1.out‚13374.66845703125& disk.nvme0n1.out‚13374.66845703125 disk.nvme1n1.in‚0 cpu‚25.618206103878094# disk./.usageGB‚1169.00732421875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1n1.out‚4571.68603515625" proc.memory.rssMB‚ 2974.2109375 disk.dm-0.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚4571.68603515625( proc.memory.availableMB‚ 740.62109375 proc.cpu.threads‚17 network.sent‚ 3.082688e+06) disk./.usagePercent‚76.14442756676438 disk.md0.in‚0" disk.dm-0.out‚14904.4521484375 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0$ memory_percent‚95.47960758209229) proc.memory.percent‚18.15314292907715 network.recv‚496986 disk.md0.out‚ 57.90234375N,Rž ‚(:Ç ááõÒàŽÅË disk.dm-0.in‚ 0.00390625& disk.nvme0c0n1.out‚13375.529296875 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 4817.65234375 cpu‚25.619849842379928 network.sent‚ 3.087626e+06 disk.dm-1.in‚0$ disk.nvme0n1.out‚13375.529296875$ memory_percent‚95.47960758209229 proc.cpu.threads‚17 disk.nvme0n1.in‚ 0.00390625 network.recv‚500731" disk.dm-0.out‚14904.9912109375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.md0.out‚ 58.2890625! disk.dm-1.out‚3164.0205078125 disk.rbd0.in‚0 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 4817.65234375 disk.md0.in‚0( proc.memory.availableMB‚ 740.62109375" proc.memory.rssMB‚ 2974.2109375) proc.memory.percent‚18.15314292907715) disk./.usagePercent‚76.14442632439432% disk./.usageGB‚1169.0073051452637;²(µœŸ ‚(:‘ ááõÒí±•Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.458' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.81933333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Å_Ê2  - _wandbruntime‚1826  _runtime‚1826‘?ýìÇ¡ Á _runtime‚1826.006762146 _step‚165 & lr-AdamW/pg1‚0.0006038196891513258 & lr-AdamW/pg2‚0.0006038196891513258 trainer/global_step‚2049 % _timestamp‚1.7845086524311638e+09+ßj¾Ì¢ Æ & lr-AdamW/pg1‚0.0006038196891513258 & lr-AdamW/pg2‚0.0006038196891513258 trainer/global_step‚2049 % _timestamp‚1.7845086524311638e+09 _runtime‚1826.006762146 _step‚165¥T²Æ€2£ - _wandbruntime‚1826  _runtime‚1826°÷'òÝ'¤ ×O H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313710023820800 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8492992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314252122459188 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313710040860160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70655630924288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2380662003200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚1826.007567498 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316090685824000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4911826 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313710023820800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚311874711493296 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20275426 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 _step‚166 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20275426 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2395052953600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2377070404618 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316091044287488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2381003427328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16593593 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 $ _timestamp‚1.784508652431961e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2380662003200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16593593 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2377410965892 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3580896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2394953975808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314251764858554 trainer/global_step‚2049 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313710023820800 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8492992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314252122459188 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313710040860160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70655630924288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2380662003200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚1826.007567498 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316090685824000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4911826 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313710023820800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚311874711493296 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20275426 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 _step‚166 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20275426 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2395052953600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2377070404618 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316091044287488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2381003427328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16593593 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 $ _timestamp‚1.784508652431961e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2380662003200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16593593 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2377410965892 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3580896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2394953975808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314251764858554 trainer/global_step‚2049¦È+m_2¦ - _wandbruntime‚1826  _runtime‚1826¡De6„§ ãM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚2049 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8496959 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16601386 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8496682 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3582630 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚312022142160768 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313858298322944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312022125121408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845086530540824e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316240428895744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313858315362304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20295239 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313858315362304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2381772102144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313858298322944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2378178792870 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2378519354256 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20295239 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36896627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314400303914278 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16601388 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2381772102144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36886468 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314400661515024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4914329 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68293857218560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316240428895744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2396063909888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316240070425088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3582627 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36896627 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20285082 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4914055 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚1826.629861154 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36886468¡Rî&¨ èM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313858298322944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2378178792870 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2378519354256 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20295239 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36896627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314400303914278 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16601388 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2381772102144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36886468 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314400661515024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4914329 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68293857218560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316240428895744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2396063909888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316240070425088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3582627 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36896627 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20285082 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4914055 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚1826.629861154 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36886468 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚2049 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8496959 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16601386 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8496682 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3582630 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚312022142160768 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313858298322944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312022125121408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845086530540824e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316240428895744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313858315362304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20295239 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313858315362304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2381772102144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232§-‡àä2© - _wandbruntime‚1826  _runtime‚1826S—Oéª ‚ 2loss/SRqqt2/assignment_loss‚0.6298924567130194 3loss/FBt2/assignment_loss‚0.0005007567750195833 %loss/total_loss‚9.266131155294456 _step‚168 1loss/SRqqt1/assignment_loss‚0.530687600688437 .loss/FRt1/detection_loss‚0.598409833492095 0loss/FBt1/detection_loss‚0.42744095471036964 epoch‚5 3loss/FBt1/assignment_loss‚0.0004398039294521344 /loss/FRt2/detection_loss‚1.6341337572347294 1loss/SRqqt2/detection_loss‚0.6960907968821832 .loss/FBt2/detection_loss‚1.009627281549446 $ _timestamp‚1.784508653055174e+09 _runtime‚1826.630484415 0loss/FRt1/assignment_loss‚1.9139133803779746 0loss/FRt2/assignment_loss‚0.9282173277215755 1loss/SRqqt1/detection_loss‚0.8967772052201547 trainer/global_step‚2049-€e@ « ‡ 0loss/FBt1/detection_loss‚0.42744095471036964 epoch‚5 3loss/FBt1/assignment_loss‚0.0004398039294521344 /loss/FRt2/detection_loss‚1.6341337572347294 1loss/SRqqt2/detection_loss‚0.6960907968821832 .loss/FBt2/detection_loss‚1.009627281549446 $ _timestamp‚1.784508653055174e+09 _runtime‚1826.630484415 0loss/FRt1/assignment_loss‚1.9139133803779746 0loss/FRt2/assignment_loss‚0.9282173277215755 1loss/SRqqt1/detection_loss‚0.8967772052201547 trainer/global_step‚2049 2loss/SRqqt2/assignment_loss‚0.6298924567130194 3loss/FBt2/assignment_loss‚0.0005007567750195833 %loss/total_loss‚9.266131155294456 _step‚168 1loss/SRqqt1/assignment_loss‚0.530687600688437 .loss/FRt1/detection_loss‚0.598409833492095¨ªd:[¬ ‚(:Ð ðáõÒƒÀÒË network.sent‚ 3.105273e+06) disk./.usagePercent‚76.14442781523837% disk./.usageGB‚1169.0073280334473 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0' proc.memory.availableMB‚ 740.5859375) proc.memory.percent‚18.15314292907715 proc.cpu.threads‚17 disk.md0.out‚ 58.48828125" disk.dm-0.out‚14906.8779296875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚13377.38623046875& disk.nvme0n1.out‚13377.38623046875' disk.nvme1c1n1.out‚4818.23193359375 network.recv‚503999 disk.md0.in‚0 disk.nvme0n1.in‚ 0.00390625" proc.memory.rssMB‚ 2974.2109375 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0! disk.dm-1.out‚3164.1298828125 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚4818.23193359375$ memory_percent‚95.47982215881348 cpu‚25.62320619909067c—¦@ž­ ‚(:“ ðáõÒµ¬–Ì gpu.0.gpu‚89.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚267.571' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.19033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ivƒ[® ‚(:Ð ÿáõÒÀ¤”Ë network.sent‚ 3.15477e+06 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚4818.95458984375 proc.cpu.threads‚17( disk./.usagePercent‚76.1444914245854% disk./.usageGB‚1169.0083045959473( disk.nvme0c0n1.out‚13378.38623046875' disk.nvme1c1n1.out‚4818.95458984375 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625) proc.memory.percent‚18.15314292907715 disk.rbd0.out‚0 disk.nvme1n1.in‚0" disk.dm-0.out‚14907.8115234375! disk.dm-1.out‚3164.1923828125$ memory_percent‚95.47984600067139( proc.memory.availableMB‚ 740.58203125" proc.memory.rssMB‚ 2974.2109375 cpu‚25.626794277780057 network.recv‚508334 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚13378.38623046875 disk.md0.out‚ 58.82421875 disk.dm-1.in‚0dÆ_±‘¯ ‚(:† ÿáõÒøŽçË gpu.0.gpu‚82.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.08' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.36 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ô?W° ‚(:Ì ŽâõÒ©ç¾Ë disk.nvme1n1.in‚0 cpu‚25.630301454958488 proc.cpu.threads‚17 network.sent‚ 3.161444e+06) disk./.usagePercent‚76.14448670357918 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0" disk.nvme1n1.out‚ 4819.91796875 disk.md0.in‚0 disk.md0.out‚ 59.3671875$ memory_percent‚95.47984600067139" disk.dm-1.out‚3164.30517578125 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme1c1n1.out‚ 4819.91796875 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚14909.6416015625" proc.memory.rssMB‚ 2974.2109375) proc.memory.percent‚18.15314292907715 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚13380.52001953125( proc.memory.availableMB‚ 740.58203125 network.recv‚511431% disk./.usageGB‚1169.0082321166992 disk.rbd0.out‚0( disk.nvme0c0n1.out‚13380.52001953125Ã&ž± ‚(:“ ŽâõÒ’ØûË gpu.0.gpu‚47.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚239.876' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚79.95866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º 5 2² - _wandbruntime‚1870  _runtime‚1870Üæ”Vdz Á & lr-AdamW/pg1‚0.0006031669004562907 & lr-AdamW/pg2‚0.0006031669004562907 trainer/global_step‚2099 % _timestamp‚1.7845086965820503e+09 _runtime‚1870.157659063 _step‚169 ƒåÌ´ Æ & lr-AdamW/pg1‚0.0006031669004562907 & lr-AdamW/pg2‚0.0006031669004562907 trainer/global_step‚2099 % _timestamp‚1.7845086965820503e+09 _runtime‚1870.157659063 _step‚169©AË_2µ - _wandbruntime‚1870  _runtime‚1870¾‡É”Ý'¶ ×O XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2450842456576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 $ _timestamp‚1.784508696583153e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37743321 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2436572529152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚2099 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321123748928000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69919690452480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚321123765967360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5026984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323559980033024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37753466 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3667596 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321123765967360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5027252 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16983245 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321679139046196 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20770221 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3667593 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319246244866896 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323559980033024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72370650499072 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8694848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚1870.158729075 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2436231105024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321678781445562 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321123748928000 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8694577 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16983243 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20770221 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323560338496512 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20760078 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16983243 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20760078 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2432894179300 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72370532909056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2436231105024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319246227827536 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323559980033024 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72370650499072 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8694848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚1870.158729075 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2436231105024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321678781445562 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321123748928000 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8694577 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16983243 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20770221 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323560338496512 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20760078 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16983243 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20760078 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2432894179300 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72370532909056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2436231105024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319246227827536ªÃ?n2¸ - _wandbruntime‚1870  _runtime‚1870mm$²ç&¹ áM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37770929 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2437682635264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8698562 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845086972046847e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321272023430144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚323709723104768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚319393675534368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚323709723104768 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3669330 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72404945389568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2099 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37760770 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20779891 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8698823 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319393658495008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321827320501286 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2437682635264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 1870.78044488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321272023430144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20769734 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16991036 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2437341203968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37770929 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69952870722560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2437341203968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321272040469504 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37770929 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69952870722560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2437341203968 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321272040469504 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37770929 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2437682635264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8698562 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845086972046847e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321272023430144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚323709723104768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚319393675534368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚323709723104768 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3669330 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72404945389568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2099 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37760770 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20779891 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8698823 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319393658495008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321827320501286 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2437682635264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 1870.78044488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321272023430144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20769734«·HŠ$2» - _wandbruntime‚1870  _runtime‚1870/üÛª¼ ‚ _runtime‚1870.781065081 0loss/FRt2/assignment_loss‚1.0156645283420036 trainer/global_step‚2099 2loss/SRqqt1/assignment_loss‚0.5755932535298274 2loss/FBt2/assignment_loss‚0.009335324731566801 /loss/FRt2/detection_loss‚1.7789490803917671 1loss/SRqqt2/detection_loss‚0.7302283659513455 %loss/total_loss‚9.505478784166527 epoch‚5 _step‚172 0loss/FRt1/assignment_loss‚1.9742581154200982 3loss/FBt1/assignment_loss‚0.0046461727844585386 1loss/SRqqt1/detection_loss‚0.7313230373537402 .loss/FBt2/detection_loss‚1.071941627353868 % _timestamp‚1.7845086972057638e+09 2loss/SRqqt2/assignment_loss‚0.5781964923911076 /loss/FRt1/detection_loss‚0.5300151466061754 .loss/FBt1/detection_loss‚0.505327639310568tʈ‰ ½ ‡ _runtime‚1870.781065081 0loss/FRt2/assignment_loss‚1.0156645283420036 trainer/global_step‚2099 2loss/SRqqt1/assignment_loss‚0.5755932535298274 2loss/FBt2/assignment_loss‚0.009335324731566801 /loss/FRt2/detection_loss‚1.7789490803917671 1loss/SRqqt2/detection_loss‚0.7302283659513455 %loss/total_loss‚9.505478784166527 epoch‚5 _step‚172 0loss/FRt1/assignment_loss‚1.9742581154200982 3loss/FBt1/assignment_loss‚0.0046461727844585386 1loss/SRqqt1/detection_loss‚0.7313230373537402 .loss/FBt2/detection_loss‚1.071941627353868 % _timestamp‚1.7845086972057638e+09 2loss/SRqqt2/assignment_loss‚0.5781964923911076 /loss/FRt1/detection_loss‚0.5300151466061754 .loss/FBt1/detection_loss‚0.505327639310568¬êyžÕW¾ ‚(:Ì âõÒºðÒË) disk./.usagePercent‚76.14447129819044 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.md0.out‚ 59.76171875" disk.dm-0.out‚14910.2861328125" disk.dm-1.out‚3164.34423828125 disk.rbd0.in‚0) proc.memory.percent‚18.15314292907715 network.sent‚ 3.177251e+06 network.recv‚514847( disk.nvme0c0n1.out‚13381.46728515625 disk.nvme1n1.in‚0 disk.dm-1.in‚0$ memory_percent‚95.48006057739258& proc.memory.availableMB‚ 740.546875" proc.memory.rssMB‚ 2974.2109375& disk.nvme0n1.out‚13381.46728515625 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0 disk.md0.in‚0% disk./.usageGB‚1169.0079956054688 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme1c1n1.out‚4820.509765625# disk.nvme1n1.out‚4820.509765625 cpu‚25.63230400920999 proc.cpu.threads‚17Ý‚`ž¿ ‚(:“ âõÒ°ÿ‘Ì gpu.0.gpu‚89.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚258.092' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.03066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õªõ ZÀ ‚(:Ï ¬âõÒ¨ÔÀË network.sent‚ 3.226775e+06) disk./.usagePercent‚76.14449217000742$ disk./.usageGB‚1169.008316040039 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚4820.92626953125% disk.nvme1n1.out‚4820.92626953125 disk.md0.in‚0" disk.dm-0.out‚14911.0830078125 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚95.48008441925049 disk.dm-1.out‚ 3164.43359375( disk.nvme0c0n1.out‚13382.39599609375( proc.memory.availableMB‚ 740.54296875) proc.memory.percent‚18.15314292907715 proc.cpu.threads‚17 network.recv‚518780& disk.nvme0n1.out‚13382.39599609375 disk.md0.out‚ 59.96484375" proc.memory.rssMB‚ 2974.2109375 cpu‚25.631004469917624=Ix—žÁ ‚(:“ ¬âõÒ»œÌ gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚280.204' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.40133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö|@|M ‚(: »âõÒœýÚË network.recv‚522193 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme0n1.out‚13384.9384765625 disk.nvme1n1.in‚0 disk.md0.in‚0! disk.dm-0.out‚14913.330078125$ memory_percent‚95.48015594482422$ disk./.usageGB‚1169.008430480957 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚4821.7490234375 disk.md0.out‚60.46875% proc.memory.availableMB‚ 740.53125) proc.memory.percent‚18.15314292907715 cpu‚25.629845837860877 proc.cpu.threads‚17 network.sent‚ 3.233455e+06) disk./.usagePercent‚76.14449962422778 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625 disk.dm-0.in‚ 0.00390625" proc.memory.rssMB‚ 2974.2109375 disk.dm-1.in‚0 disk.dm-1.out‚ 3164.484375' disk.nvme0c0n1.out‚13384.9384765625$ disk.nvme1n1.out‚4821.7490234375ësiK~à ‚(:“ »âõÒÊÛÌ gpu.0.gpu‚79.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoº­YP ryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚278.217' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.73899999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dM÷b2Ä - _wandbruntime‚1913  _runtime‚1913*ÈKÇÅ Á & lr-AdamW/pg1‚0.0006024951052319986 & lr-AdamW/pg2‚0.0006024951052319986 trainer/global_step‚2149 % _timestamp‚1.7845087398622403e+09 _runtime‚1913.438359618 _step‚173ùÁÄóÌÆ Æ _step‚173 & lr-AdamW/pg1‚0.0006024951052319986 & lr-AdamW/pg2‚0.0006024951052319986 trainer/global_step‚2149 % _timestamp‚1.7845087398622403e+09 _runtime‚1913.438359618­:n(2Ç - _wandbruntime‚1913  _runtime‚1913»`Èß'È ÙO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2506713443328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚328537491074560 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21244730 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3754296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚2149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38617623 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2492141630976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74085541281280 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚326617761201136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17372895 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71578722568704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚331029632705536 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331029274242048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71578703956480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329105798032570 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2491800206848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1913.438566758 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331029632705536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2491800206848 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38627768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17372893 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5141426 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74085417399808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚326617778240496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331029274242048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328537474035200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17372893 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2492141630976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17372895 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8896005 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38617623 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2492141630976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74085541281280 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚326617761201136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17372895 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71578722568704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚331029632705536 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331029274242048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71578703956480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329105798032570 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2491800206848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1913.438566758 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331029632705536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2491800206848 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38627768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17372893 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5141426 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74085417399808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚326617778240496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331029274242048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328537474035200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17372893 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2492141630976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17372895 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496®¢ý)2Ê - _wandbruntime‚1914  _runtime‚19141ùúé&Ë ãM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _runtime‚1914.372195102 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38635072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2507936045056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17380688 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329254694689040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38645231 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5143720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71611884226560 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3756030 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331178658843136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2492910305792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328685748537344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17380688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21264543 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚326765208907968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21254386 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2493251737088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚2149 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21264543 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38645231 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17380686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2489145219686 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8900041 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2507822394368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚331179017313792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8899747 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291ãÙÐ+ÅÌ èM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚331179017313792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8899747 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _runtime‚1914.372195102 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38635072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2507936045056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17380688 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚329254694689040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMoniʘœ¼)!tor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38645231 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5143720 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71611884226560 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3756030 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331178658843136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2492910305792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328685748537344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17380688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21264543 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚326765208907968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21254386 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2493251737088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚2149 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21264543 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38645231 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17380686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2489145219686 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8900041 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2507822394368 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2547710732800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17762545 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21739525 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39502070 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3840996 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2547710732800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9097755 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17762545 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21729382 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚335951199142400 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2547369308672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73237717460480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚335951216181760 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9097461 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5256759 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚336532814619578 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2543520044842 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚338498568451072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845087831668613e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2562676167680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2543860606116 trainer/global_step‚2199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5256468 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 _step‚178 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338498926914560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338498568451072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2547369308672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚338498926914560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5256759 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚336532814619578 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2543520044842 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚338498568451072 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845087831668613e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2562676167680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2543860606116 trainer/global_step‚2199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5256468 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 _step‚178 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338498926914560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338498568451072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2547369308672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚338498926914560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2547710732800 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17762545 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21739525 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39502070 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3840996 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2547710732800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528²øºd2Ü - _wandbruntime‚1957  _runtime‚1957öÁP[è&Ý âM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338648311522816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338647953052160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39519533 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3842727 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2544968994480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚302 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39519533 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336099490683904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2548479407616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17770336 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9101715 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17770338 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2563686765056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚2199 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21749195 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3842730 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2548820838912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39509374 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚1957.365938638 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39509374 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚75834584495616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336099473644544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17770336 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334136742281568 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21739038 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5258686 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2563796221440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2548479407616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9101413 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75834712564224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚179 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17770338 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2544628433094 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0_i+Ùí&Þ çM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚336681353675302 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334136725242208 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5258985 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21749195 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336099473644544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21749195 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3842730 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2548820838912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39509374 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚1957.365938638 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39509374 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚75834584495616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336099473644544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17770336 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334136742281568 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21739038 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5258686 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2563796221440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2548479407616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9101413 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75834712564224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚179 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17770338 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2544628433094 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338648311522816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338647953052160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39519533 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3842727 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2544968994480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚302 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39519533 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336099490683904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2548479407616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17770336 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9101715 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17770338 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2563686765056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚2199 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5370904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚343959831206586 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2602938410496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3927696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5371180 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345967862660096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343364924249600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3927693 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345967862660096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2618402309120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18152193 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343960188807220 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚2249 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2599003258250 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343364924249600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2599343819524 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22214034 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2602938410496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2603279834624 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9298876 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40376372 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚182 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77515133273600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚341360827948336 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343364941288960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344mãÚä'í ÞO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚341360844987696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2603279834624 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345967862660096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343364924249600 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3927693 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345967862660096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2618402309120 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18152193 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343960188807220 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚2249 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2599003258250 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343364924249600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2599343819524 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22214034 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2602938410496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2603279834624 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9298876 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40376372 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚182 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77515133273600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚341360827948336 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343364941288960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22224177 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77515255057920 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚345968221123584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9298597 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5370904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚343959831206586 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2602938410496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3927696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5371180 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚346117247261184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2619517375488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚343513215791104 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5373427 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40393835 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3929430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3929427 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2604389940736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22233847 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2604389940736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5373146 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2600111646502 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845088272133539e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40393835 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40383676 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18159986 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22223690 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341508258615808 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343513198751744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22223690 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2604048509440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40383676 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚344108370262310 trainer/global_step‚2249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2600452207888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚344108727863056 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341508275655168 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18159988 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9302573 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9302573 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚346117247261184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2619517375488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚343513215791104 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5373427 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40393835 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3929430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3929427 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2604389940736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22233847 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2604389940736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5373146 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2600111646502 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845088272133539e+09 G:DeviceStatsMoniLâ<ì tor.on_train_batch_end/active.all.allocated‚40393835 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40383676 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18159986 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22223690 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341508258615808 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343513198751744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22223690 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2604048509440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40383676 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚344108370262310 trainer/global_step‚2249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2600452207888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚344108727863056 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341508275655168 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18159988 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2658507512320 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79229977204224 _runtime‚2043.377862182 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22708829 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2658848936448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348732361321936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚350778666396160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚350778666396160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79230101085696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76555744468480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2674232735744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2654486471658 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22698686 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5486444 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353437156869120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2658848936448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4014393 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353437515332608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353437156869120 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2658507512320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4014396 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚186 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7845088698023703e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2658507512320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4014396 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚186 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7845088698023703e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2658507512320 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79229977204224 _runtime‚2043.377862182 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22708829 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2658848936448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348732361321936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚350778666396160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚350778666396160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79230101085696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76555744468480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2674232735744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2654486471658 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22698686 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5486444 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353437156869120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2658848936448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4014393 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353437515332608 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353437156869120 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224ºüðg52€ - _wandbruntime‚2043  _runtime‚2043ñ»a„  âM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚187 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22718499 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79264272427008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9504867 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 $ _timestamp‚1.784508870424101e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18549636 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41257978 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18549636 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2659959042560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351535744450064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚348879791989408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚350926940898304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22718499 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4016127 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2675347688448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353586899940864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18549638 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚76588924738560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2655935421296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41257978 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚350926923858944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41268137 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2659617611264 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22708342 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4016130 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2043.999828826 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2675457144832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348879809028768 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2655594859910 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41268137 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18549638 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353586899940864 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2659959042560 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351535386849318 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9504574 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344í!í&‚ çM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351535744450064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚348879791989408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚350926940898304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22718499 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4016127 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2675347688448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353586899940864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18549638 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚76588924738560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2655935421296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41257978 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚350926923858944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41268137 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2659617611264 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22708342 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4016130 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2043.999828826 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2675457144832 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348879809028768 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2655594859910 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41268137 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18549638 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353586899940864 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2659959042560 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351535386849318 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9504574 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚187 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22718499 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79264272427008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9504867 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 $ _timestamp‚1.784508870424101e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18549636 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41257978 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18549636 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2659959042560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360»¸´Þ2ƒ - _wandbruntime‚2044  _runtime‚2044áí{v„ € $ _timestamp‚1.784508870425153e+09 _step‚188 /loss/FRt2/assignment_loss‚0.866480458027138 1loss/FBt1/assignment_loss‚0.04107891111812963 3loss/FBt2/assignment_loss‚0.0009005870542084437 /loss/FRt2/detection_loss‚1.6169346527436192 1loss/SRqqt1/detection_loss‚0.7677027322153395 1loss/SRqqt2/detection_loss‚0.7130823522132386 /loss/FBt1/detection_loss‚0.6212727800281026 epoch‚5 1loss/SRqqt2/assignment_loss‚0.588595589021202 /loss/FRt1/detection_loss‚0.5874131309446395 %loss/total_loss‚8.860543351951643 trainer/global_step‚2299 _runtime‚2044.000416537 0loss/FRt1/assignment_loss‚1.9307980445807817 2loss/SRqqt1/assignment_loss‚0.5475252095279534 /loss/FBt2/detection_loss‚0.5787589044772904ç£D … … _step‚188 /loss/FRt2/assignment_loss‚0.866480458027138 1loss/FBt1/assignment_loss‚0.04107891111812963 3loss/FBt2/assignment_loss‚0.0009005870542084437 /loss/FRt2/detection_loss‚1.6169346527436192 1loss/SRqqt1/detection_loss‚0.7677027322153395 1loss/SRqqt2/detection_loss‚0.7130823522132386 /loss/FBt1/detection_loss‚0.6212727800281026 epoch‚5 1loss/SRqqt2/assignment_loss‚0.588595589021202 /loss/FRt1/detection_loss‚0.5874131309446395 %loss/total_loss‚8.860543351951643 trainer/global_step‚2299 _runtime‚2044.000416537 0loss/FRt1/assignment_loss‚1.9307980445807817 2loss/SRqqt1/assignment_loss‚0.5475252095279534 /loss/FBt2/detection_loss‚0.5787589044772904 $ _timestamp‚1.784508870425153e+09¼v_S)W† ‚(:Ì ÑãõÒë˜ôË network.sent‚ 3.464805e+06 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚7102.3896484375 disk.md0.in‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚14208.15380859375 cpu‚25.617781650531132 proc.cpu.threads‚17% disk./.usageGB‚1169.0070991516113 disk.md0.out‚ 64.78515625# disk.dm-0.out‚18006.92626953125 disk.rbd0.out‚0$ memory_percent‚96.98371887207031" proc.memory.rssMB‚ 2974.2109375 disk.dm-1.out‚3165.451171875 network.recv‚558219) disk./.usagePercent‚76.14441290679767 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0$ proc.memory.availableMB‚494.1875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚14208.15380859375& disk.nvme1c1n1.out‚7102.3896484375) proc.memory.percent‚18.15314292907715Q üS“‡ ‚(:ˆ ÑãõÒëÜøÌ gpu.0.gpu‚90.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚272.937' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚90.979 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²(¶Xˆ ‚(:Í àãõÒæ¯Ë disk.dm-1.out‚ 3165.5390625 disk.rbd0.out‚0$ memory_percent‚96.98386192321777 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625% disk.nvme1n1.out‚7103.69970703125# disk.dm-0.out‚18007.93017578125 disk.dm-1.in‚0% disk./.usageGB‚1168.5314712524414% disk.nvme0n1.out‚14209.7861328125' disk.nvme1c1n1.out‚7103.69970703125 disk.md0.out‚ 65.69140625 disk.rbd0.in‚0' proc.memory.availableMB‚ 494.1640625) proc.memory.percent‚18.15314292907715 proc.cpu.threads‚17 network.recv‚562022) disk./.usagePercent‚76.11343242158875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625" proc.memory.rssMB‚ 2974.2109375 cpu‚25.619465044558545 network.sent‚ 3.514334e+06' disk.nvme0c0n1.out‚14209.7861328125 disk.md0.in‚0¶ÏNž‰ ‚(:“ àãõÒÉÁûË gpu.0.gpu‚94.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚259.801' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.60033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ¤´&WŠ ‚(:Ì ïãõÒ£êÞË disk.rbd0.out‚0 disk.md0.in‚0# memory_percent‚96.9839334487915 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚14221.91748046875 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625( proc.memory.availableMB‚ 494.15234375 proc.cpu.threads‚17 network.sent‚ 3.520998e+06( disk./.usagePercent‚76.1134354032769 disk.rbd0.in‚0& disk.nvme1c1n1.out‚7105.3681640625 disk.nvme1n1.in‚0# disk.dm-0.out‚18019.79541015625" proc.memory.rssMB‚ 2974.2109375) proc.memory.percent‚18.15314292907715% disk./.usageGB‚1168.5315170288086 disk.dm-1.out‚ 3165.57421875& disk.nvme0n1.out‚14221.91748046875$ disk.nvme1n1.out‚7105.3681640625 disk.md0.out‚ 66.60546875 cpu‚25.61991158037245 network.recv‚565830Õ0ž‹ ‚(:“ ïãõÒ“¬›Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚288.647' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.21566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80944817056256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358192391503360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9702955 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2709969685066 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23193481 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5601859 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4101093 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360906451078144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚356103911734896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2714076614144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2714418038272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78214776584704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845089129589074e+09 _step‚190 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23183338 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78214757972480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358813864380602 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360906451078144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23193481 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42114831 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358814221981236 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool‡?rÿ£.current‚272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180#y©âä'‘ ÞO ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80944940937728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚2349 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚358192391503360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18931493 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80944817056256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358192391503360 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9702955 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2709969685066 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23193481 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5601859 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4101093 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360906451078144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚356103911734896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2714076614144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2714418038272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78214776584704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845089129589074e+09 _step‚190 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23183338 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78214757972480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358813864380602 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360906451078144 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23193481 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42114831 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚358814221981236 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180¾g'º2’ - _wandbruntime‚2087  _runtime‚2087QÔHé&“ ãM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚361055835679232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _step‚191 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2711078073318 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358340648966144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2731179500032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚356251342402368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5604109 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4102830 _runtime‚2087.142224314 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18939286 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23192994 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78247938242560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358962403436326 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23192994 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9706652 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18939288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42142439 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚356251325363008 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚361056194149888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5603825 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845089135664933e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2715528144384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚358962761037072 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42132280 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9706939 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42142439 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42132280 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18939288 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23203151 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358340648966144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78247956854784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80979117742592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2715186713088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358340666005504 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80979239519744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2711418634704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23203151 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2715186713088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2731282664960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528ÿÍn0î&” èM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4102830 _runtime‚2087.142224314 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18939286 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23192994 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78247938242560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358962403436326 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23192994 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9706652 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18939288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42142439 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚356251325363008 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚361056194149888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5603825 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845089135664933e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2715528144384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚358962761037072 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42132280 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9706939 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42142439 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42132280 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18939288 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23203151 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚358340648966144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78247956854784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80979117742592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2715186713088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358340666005504 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80979239519744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2711418634704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23203151 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2715186713088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2731282664960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚361055835679232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _step‚191 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2711078073318 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358340648966144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2731179500032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚356251342402368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5604109 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374¿œØM 2• - _wandbruntime‚2087  _runtime‚20878úã– ü _runtime‚2087.146254939 _step‚192 1loss/FBt2/assignment_loss‚0.00467751445614163 $loss/total_loss‚8.87569847471864 epoch‚5 .loss/FBt2/detection_loss‚0.643963904456339 trainer/global_step‚2349 0loss/FRt1/assignment_loss‚2.0010818250222466 2loss/SRqqt2/assignment_loss‚0.5603029941230758 .loss/FRt1/detection_loss‚0.503136874592254 /loss/FRt2/detection_loss‚1.4859741748531454 1loss/SRqqt2/detection_loss‚0.7058535542552946 0loss/FRt2/assignment_loss‚1.0025082258834053 2loss/SRqqt1/assignment_loss‚0.5506119937586029 1loss/FBt1/assignment_loss‚0.01734499445801838 1loss/SRqqt1/detection_loss‚0.7899612710835778 .loss/FBt1/detection_loss‚0.610281147776538 $ _timestamp‚1.784508913570908e+09]'/2—  1loss/SRqqt1/detection_loss‚0.7899612710835778 .loss/FBt1/detection_loss‚0.610281147776538 $ _timestamp‚1.784508913570908e+09 _runtime‚2087.146254939 _step‚192 1loss/FBt2/assignment_loss‚0.00467751445614163 $loss/total_loss‚8.87569847471864 epoch‚5 .loss/FBt2/detection_loss‚0.643963904456339 trainer/global_step‚2349 0loss/FRt1/assignment_loss‚2.0010818250222466 2loss/SRqqt2/assignment_loss‚0.5603029941230758 .loss/FRt1/detection_loss‚0.503136874592254 /loss/FRt2/detection_loss‚1.4859741748531454 1loss/SRqqt2/detection_loss‚0.7058535542552946 0loss/FRt2/assignment_loss‚1.0025082258834053 2loss/SRqqt1/assignment_loss‚0.5506119937586029 1loss/FBt1/assignment_loss‚0.01734499445801838À@™Âö˜ ‚(:Ð þãõÒ¸šË proc.cpu.threads‚17 network.recv‚569173$ disk.nvme1n1.out‚7106.2646484375 disk.md0.in‚0 disk.md0.out‚ 67.03515625 disk.rbd0.out‚0 network.sent‚ 3.524063e+06% disk./.usageGB‚1168.5309562683105 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚7106.2646484375$ memory_percent‚96.98421955108643( proc.memory.availableMB‚ 494.10546875 cpu‚25.618602716740007 disk.nvme1n1.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚14222.8134765625" proc.memory.rssMB‚ 2974.2109375) disk./.usagePercent‚76.11339887759718 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚18020.35791015625" disk.dm-1.out‚3165.88720703125 ̳½e disk.rbd0.in‚0' disk.nvme0c0n1.out‚14222.8134765625) proc.memory.percent‚18.15314292907715 ¹œ¬“™ ‚(:ˆ þãõÒçÄ„Ì gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.914' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.638 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑÀZš ‚(:Ï äõÒÞ™©Ë disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚7108.80126953125 disk.md0.in‚0 disk.md0.out‚ 67.51171875) proc.memory.percent‚18.15314292907715 network.sent‚ 3.588293e+06 network.recv‚573129) disk./.usagePercent‚76.11339788370111 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚7108.80126953125 disk.dm-0.in‚ 0.00390625$ memory_percent‚96.98443412780762 disk.nvme1c1n1.in‚0" proc.memory.rssMB‚ 2974.2109375 proc.cpu.threads‚17 disk.dm-1.in‚0" disk.dm-1.out‚3165.95751953125' disk.nvme0c0n1.out‚14237.8408203125% disk.nvme0n1.out‚14237.8408203125" disk.dm-0.out‚18036.8408203125' proc.memory.availableMB‚ 494.0703125 cpu‚25.61741312109856% disk./.usageGB‚1168.5309410095215ô*FÏž› ‚(:“ äõÒòæîË gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚278.315' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.77166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡]2œ - _wandbruntime‚2129  _runtime‚2129ÂÌ<$Ç Á & lr-AdamW/pg2‚0.0005988525839260012 trainer/global_step‚2399 % _timestamp‚1.7845089554555461e+09 _runtime‚2129.031125865 _step‚193 & lr-AdamW/pg1‚0.0005988525839260012<ßÚÌž Æ & lr-AdamW/pg2‚0.0005988525839260012 trainer/global_step‚2399 % _timestamp‚1.7845089554555461e+09 _runtime‚2129.031125865 _step‚193 & lr-AdamW/pg1‚0.0005988525839260012Áóe2Ÿ - _wandbruntime‚2129  _runtime‚2129ãwJß'  ÙO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9904037 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79873790088704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79873771476480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19321143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368376103750656 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19321145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23667990 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9903732 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19321145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2769987140096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚365606116610560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368376103750656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2765452898474 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚365606099571200 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42999278 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2785992494080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368375745287168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82659652409856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚365606116610560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82659782582784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2129.035023002 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23678133 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2769645715968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368375745287168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚366241238568244 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19321143 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845089554564784e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42989133 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366240880967610 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚363475428069136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚363475445108496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23667990 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚365606099571200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚305 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4187796 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5715939 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2399 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2769987140096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0„[•ôä'¡ ÞO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79873790088704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79873771476480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19321143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368376103750656 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19321145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23667990 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9903732 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19321145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2769987140096 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚365606116610560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368376103750656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2765452898474 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚365606099571200 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42999278 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2785992494080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368375745287168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82659652409856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚365606116610560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82659782582784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚2129.035023002 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23678133 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2769645715968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368375745287168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚366241238568244 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19321143 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845089554564784e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42989133 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚366240880967610 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚363475428069136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚363475445108496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23667990 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚365606099571200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚305 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4187796 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5715939 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2399 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2769987140096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9904037 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180·óå2¢ - _wandbruntime‚2129  _runtime‚2129þ¸è™Ð&£ ãM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43016741 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365754374073344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19328938 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5718245 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚363622858736608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2766901848112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19328936 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚368525488358912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4189530 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚365754374073344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366389777624080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2771097246208 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82694078153728 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _step‚195 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9907772 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368525488358912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚2129.641839912 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43016741 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368525129888256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365754391112704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚365754391112704 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2771097246208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79906951746560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2770755814912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2766561286726 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43006582 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366389420023334 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23687803 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2787107794944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19328938 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19328936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5718530 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.s¡Wûümall_pool.allocated‚272´Ê,±î&¤ èM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2771097246208 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82694078153728 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _step‚195 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9907772 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368525488358912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚2129.641839912 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43016741 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368525129888256 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365754391112704 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚365754391112704 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2771097246208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79906951746560 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2770755814912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2766561286726 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43006582 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚366389420023334 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23687803 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2787107794944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19328938 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19328936 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5718530 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43016741 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365754374073344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19328938 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5718245 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚363622858736608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2766901848112 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19328936 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚368525488358912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4189530 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚365754374073344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚366389777624080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180ÃL22¥ - _wandbruntime‚2129  _runtime‚2129€51ù¦  /loss/FRt1/detection_loss‚0.5673991425381288 1loss/SRqqt1/detection_loss‚0.8593934368564199 1loss/SRqqt2/detection_loss‚0.6610682479287707 #loss/total_loss‚8.8615819560573 trainer/global_step‚2399 0loss/FRt1/assignment_loss‚2.0424845425103255 2loss/SRqqt1/assignment_loss‚0.6390850717087726 2loss/SRqqt2/assignment_loss‚0.6036913951133245 /loss/FBt2/detection_loss‚0.5362897971399913 2loss/FBt1/assignment_loss‚0.007993821275348464 /loss/FBt1/detection_loss‚0.6343152087788486 _runtime‚2129.643136815 0loss/FRt2/assignment_loss‚0.8538784497124808 /loss/FRt2/detection_loss‚1.4413195331909117 epoch‚5 % _timestamp‚1.7845089560678365e+09 _step‚196 2loss/FBt2/assignment_loss‚0.014663309303977306ñÚìÔ § † 0loss/FRt1/assignment_loss‚2.0424845425103255 2loss/SRqqt1/assignment_loss‚0.6390850717087726 2loss/SRqqt2/assignment_loss‚0.6036913951133245 /loss/FBt2/detection_loss‚0.5362897971399913 2loss/FBt1/assignment_loss‚0.007993821275348464 /loss/FBt1/detection_loss‚0.6343152087788486 _runtime‚2129.643136815 0loss/FRt2/assignment_loss‚0.8538784497124808 /loss/FRt2/detection_loss‚1.4413195331909117 epoch‚5 % _timestamp‚1.7845089560678365e+09 _step‚196 2loss/FBt2/assignment_loss‚0.014663309303977306 /loss/FRt1/detection_loss‚0.5673991425381288 1loss/SRqqt1/detection_loss‚0.8593934368564199 1loss/SRqqt2/detection_loss‚0.6610682479287707 #loss/total_loss‚8.8615819560573 trainer/global_step‚2399Ä)Á—\Q¨ ‚(:Æ œäõÒ¾¯×Ë disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚96.98605537414551" proc.memory.rssMB‚ 2974.2109375 network.recv‚576116) disk./.usagePercent‚76.15280362570097 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚7109.32470703125 disk.rbd0.out‚0) proc.memory.percent‚18.15314292907715 cpu‚25.61623400049133" disk.nvme0n1.out‚ 14244.6328125% disk.nvme1n1.out‚7109.32470703125" disk.dm-1.out‚3166.09130859375$ disk.nvme0c0n1.out‚ 14244.6328125 network.sent‚ 3.6041e+06 disk.md0.in‚0 disk.md0.out‚ 67.7890625" disk.dm-0.out‚18043.4150390625 disk.rbd0.in‚0' proc.memory.availableMB‚ 493.8046875 proc.cpu.threads‚17% disk./.usageGB‚1169.1359176635742 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0…zX{ž© ‚(:“ œäõÒ¬¤šÌ gpu.0.gpu‚54.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚284.133' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.71099999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ†¬Rª ‚(:Ç «äõÒ܃³Ë& disk.nvme0c0n1.out‚14249.720703125 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0" disk.dm-1.out‚3166.16943359375% proc.memory.availableMB‚ 482.09375 cpu‚25.63371110420186 network.sent‚ 3.655541e+06% disk./.usageGB‚1169.1359405517578 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme0n1.out‚14249.720703125% disk.nvme1c1n1.out‚7110.095703125# disk.nvme1n1.out‚7110.095703125 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0) disk./.usagePercent‚76.15280511654504# disk.dm-0.out‚18048.30224609375 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚97.05753326416016 proc.memory.rssMB‚ 2986.09375* proc.memory.percent‚18.225669860839844 network.recv‚580195 disk.nvme0n1.in‚ 0.00390625 disk.md0.out‚ 68.20703125 proc.cpu.threads‚17‘û¾”ž« ‚(:“ «äõÒ ˆüË gpu.0.gpu‚47.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚119.927' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.97566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á£×U2¬ - _wandbruntime‚2145  _runtime‚2145^ÿô&¥=­ Ÿ{ 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7675958275794983 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03718556463718414 particle/accuracy_2_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚5 _step‚197 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6510716676712036 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.995572030544281 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6493310928344727 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6908960938453674 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_5_of_5‚0 6 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.035871721804142 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7280746102333069 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.037098679691553116 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19635125994682312 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19494445621967316 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8177706599235535 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8275510668754578 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5570285320281982 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14110638201236725 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.267706036567688 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22596336901187897 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984273314476013 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6108496785163879 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6146363615989685 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3252089023590088 *"Purity/2FRt0SRqqt1FBt/event_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9897685050964355 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0077119930647313595 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8240740299224854 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8199924230575562 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1589665412902832 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.507264256477356 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8751808404922485 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1488092690706253 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 _runtime‚2145.428237707 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20167700946331024 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_2_of_5‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08994295448064804 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6098178029060364 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6565383672714233 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14403240382671356 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7011147737503052 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.018866922706365585 particle/accuracy_1_of_1‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.722284197807312 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13920390605926514 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6485235691070557 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5593405961990356 particle/accuracy_5_of_5‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38425928354263306 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.20666666328907013 *jet/accuracy_1_of_1‚0.2798610031604767 particle/accuracy_1_of_3‚1 trainer/global_step‚2399 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8660141229629517 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17371788620948792 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 /particle/accuracy_4_of_4‚0.9425913691520691 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1281946897506714 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8240740299224854 +jet/accuracy_3_of_4‚0.09104790538549423 particle/accuracy_3_of_4‚1 jet/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10722974687814713 )loss/val_total_loss‚8.468607625387861 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963101148605347 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_2‚0.7290555238723755 particle/accuracy_3_of_3‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6490933299064636 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03431779891252518 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.037098679691553116 *jet/accuracy_1_of_4‚0.9674862623214722 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36120694875717163 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7337139844894409 +particle/sensitivity‚0.6329580545425415 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9051735401153564 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_3‚0.8970398902893066 (particle/accuracy‚0.8477839231491089 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.437447190284729 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17616257071495056 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7286475896835327 )jet/accuracy_2_of_3‚0.545346200466156 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7078295946121216 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1261715590953827 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6407679319381714 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8177706599235535 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5974581837654114 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7539647221565247 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7009899020195007 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6926953792572021 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43697166442871094 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9901471138000488 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7584338188171387 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912348985671997 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12550155818462372 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1409515142440796 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.035837460309267044 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.768028736114502 particle/accuracy_2_of_4‚1 7validation_average_jet_accuracy‚0.32882171869277954 jet/accuracy_4_of_5‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7537757754325867 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03622866049408913 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4503459334373474 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9961692094802856 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6208319067955017 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0983661562204361 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7005208730697632 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44598469138145447 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7295041084289551 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6310343742370605 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11182200163602829 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042613428086042404 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03003961406648159 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9372747540473938 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4589070677757263 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8104686737060547 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11394041031599045 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6617823839187622 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36537009477615356 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972329139709473 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6146363615989685 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8254914283752441 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 +jet/accuracy_4_of_4‚0.01908157207071781 particle/accuracy_1_of_4‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 particle/accuracy_4_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976788759231567 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6508035063743591 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09257493913173676 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5638145804405212 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014128400012850761 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6410401463508606 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6777403950691223 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.996094286441803 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9938672184944153 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1261715590953827 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6606308817863464 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212961196899414 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 % _timestamp‚1.7845089718522408e+09 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6485836505889893 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6143617033958435 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7409108281135559 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +particle/specificity‚0.9065338969230652 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12935082614421844 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6465954780578613 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12137166410684586 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.702269434928894 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20006538927555084 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07581448554992676 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6222588419914246 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967668652534485 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07865520566701889 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.110713891685009 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8242630958557129 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5558333396911621 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.14941208064556122{*’ÿª ® ¤{ 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.43697166442871094 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9901471138000488 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7584338188171387 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912348985671997 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12550155818462372 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1409515142440796 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.035837460309267044 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.768028736114502 particle/accuracy_2_of_4‚1 7validation_average_jet_accuracy‚0.32882171869277954 jet/accuracy_4_of_5‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7537757754325867 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03622866049408913 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4503459334373474 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9961692094802856 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6208319067955017 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0983661562204361 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7005208730697632 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44598469138145447 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7295041084289551 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6310343742370605 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11182200163602829 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.0426134280860z’Få342404 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03003961406648159 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9372747540473938 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4589070677757263 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8104686737060547 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11394041031599045 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6617823839187622 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36537009477615356 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972329139709473 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6146363615989685 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8254914283752441 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 +jet/accuracy_4_of_4‚0.01908157207071781 particle/accuracy_1_of_4‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 particle/accuracy_4_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976788759231567 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6508035063743591 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09257493913173676 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5638145804405212 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014128400012850761 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6410401463508606 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6777403950691223 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.996094286441803 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9938672184944153 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1261715590953827 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6606308817863464 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212961196899414 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 % _timestamp‚1.7845089718522408e+09 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6485836505889893 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6143617033958435 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7409108281135559 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +particle/specificity‚0.9065338969230652 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12935082614421844 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6465954780578613 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12137166410684586 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.702269434928894 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20006538927555084 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07581448554992676 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6222588419914246 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967668652534485 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07865520566701889 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.110713891685009 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8242630958557129 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5558333396911621 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.14941208064556122 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7675958275794983 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03718556463718414 particle/accuracy_2_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚5 _step‚197 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6510716676712036 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.995572030544281 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6493310928344727 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6908960938453674 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_5_of_5‚0 6 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.035871721804142 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7280746102333069 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.037098679691553116 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19635125994682312 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19494445621967316 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8177706599235535 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8275510668754578 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5570285320281982 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14110638201236725 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.267706036567688 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22596336901187897 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984273314476013 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6108496785163879 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6146363615989685 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3252089023590088 *"Purity/2FRt0SRqqt1FBt/event_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9897685050964355 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0077119930647313595 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8240740299224854 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8199924230575562 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1589665412902832 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.507264256477356 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8751808404922485 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1488092690706253 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 _runtime‚2145.428237707 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20167700946331024 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20000000298023224 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_2_of_5‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08994295448064804 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6098178029060364 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6565383672714233 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14403240382671356 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7011147737503052 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.018866922706365585 particle/accuracy_1_of_1‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.722284197807312 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13920390605926514 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6485235691070557 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5593405961990356 particle/accuracy_5_of_5‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.38425928354263306 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.20666666328907013 *jet/accuracy_1_of_1‚0.2798610031604767 particle/accuracy_1_of_3‚1 trainer/global_step‚2399 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8660141229629517 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17371788620948792 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 /particle/accuracy_4_of_4‚0.9425913691520691 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1281946897506714 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8240740299224854 +jet/accuracy_3_of_4‚0.09104790538549423 particle/accuracy_3_of_4‚1 jet/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10722974687814713 )loss/val_total_loss‚8.468607625387861 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963101148605347 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_2‚0.7290555238723755 particle/accuracy_3_of_3‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6490933299064636 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03431779891252518 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.037098679691553116 *jet/accuracy_1_of_4‚0.9674862623214722 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36120694875717163 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7337139844894409 +particle/sensitivity‚0.6329580545425415 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9051735401153564 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_3‚0.8970398902893066 (particle/accuracy‚0.8477839231491089 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.437447190284729 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17616257071495056 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7286475896835327 )jet/accuracy_2_of_3‚0.545346200466156 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7078295946121216 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1261715590953827 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6407679319381714 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8177706599235535 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5974581837654114 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7539647221565247 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7009899020195007 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6926953792572021ņÑ=¥V¯ ‚(:Ë ºäõÒ¦“¾Ë network.sent‚ 3.671306e+06 network.recv‚583384% disk./.usageGB‚1169.0067138671875 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚14631.33642578125# disk.dm-0.out‚18529.37255859375 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 7511.86328125( proc.memory.availableMB‚ 739.54296875 proc.memory.rssMB‚ 2987.09375 cpu‚25.632098105744564 disk.nvme1n1.in‚0 disk.md0.out‚ 68.5703125* proc.memory.percent‚18.231773376464844 proc.cpu.threads‚17) disk./.usagePercent‚76.14438781092248" disk.nvme1n1.out‚ 7511.86328125! disk.dm-1.out‚3467.7041015625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚14631.33642578125$ memory_percent‚95.48618793487549 8Ÿ<‘° ‚(:† ºäõÒ¡´ýË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.069' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.023 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ÅûU± ‚(:Ê ÉäõÒΫË disk.dm-1.in‚0* proc.memory.percent‚18.231773376464844 network.recv‚587523) disk./.usagePercent‚76.14443054845252# disk.dm-0.out‚18539.01708984375$ memory_percent‚95.48623561859131 cpu‚25.630855167313964 proc.cpu.threads‚17 network.sent‚ 3.717276e+06 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚14640.93408203125& disk.nvme0n1.out‚14640.93408203125 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 69.15234375! disk.dm-1.out‚3657.0908203125 disk.nvme0n1.in‚ 0.00390625! disk.nvme1n1.out‚ 7702.5234375 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 739.53515625 proc.memory.rssMB‚ 2987.09375% disk./.usageGB‚1169.0073699951172# disk.nvme1c1n1.out‚ 7702.5234375L®²4ž² ‚(:“ ÉäõÒȱñË gpu.0.gpu‚76.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.221' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.74033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆë»%2³ - _wandbruntime‚2188  _runtime‚2188¶\ Æ´ À & lr-AdamW/pg1‚0.0005980676981653737 & lr-AdamW/pg2‚0.0005980676981653737 trainer/global_step‚2449 $ _timestamp‚1.784509014775783e+09 _runtime‚2188.351359512 _step‚198·yÂé˵ Å trainer/global_step‚2449 $ _timestamp‚1.784509014775783e+09 _runtime‚2188.351359512 _step‚198 & lr-AdamW/pg1‚0.0005980676981653737 & lr-AdamW/pg2‚0.0005980676981653737Ærú2¶ - _wandbruntime‚2188  _runtime‚2188ÈÏÅóá'· ÛO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10154178 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19833603 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19833603 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4278612 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚372768217450960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84417822367744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚375609457411744 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2845874080256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚374962185593344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2841239960784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2862458724864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377808059673600 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24244284 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2845874080256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24244284 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚372768234490320 trainer/global_step‚2449 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚377807701210112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2845532656128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚374962185593344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚374962168553984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚377808059673600 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5875282 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚375609815012378 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚374962168553984 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4278609 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5875566 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81555363642880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2845532656128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377807701210112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2862559799808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84417942054912 % _timestamp‚1.7845090147766657e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24234141 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634[@ë¸ àO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚374962168553984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚377808059673600 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5875282 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚375609815012378 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚374962168553984 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4278609 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5875566 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81555363642880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2845532656128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377807701210112 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2862559799808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84417942054912 % _timestamp‚1.7845090147766657e+09 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24234141 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10154178 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19833603 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19833603 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4278612 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚372768217450960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84417822367744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requesteÁ¤rãd_bytes.all.freed‚375609457411744 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2845874080256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚374962185593344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2841239960784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2862458724864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377808059673600 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24244284 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2845874080256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24244284 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚372768234490320 trainer/global_step‚2449 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚377807701210112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2845532656128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚374962185593344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581ÇñÞA2¹ - _wandbruntime‚2188  _runtime‚2188¾ùÉê&º äM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 % _timestamp‚1.7845090153910942e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4280346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24243797 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚377957085811200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚84452236573184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2863579271680 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5877565 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44095350 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚372915665157792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377957444281856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚377957444281856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375110460095488 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2863674048000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚375110443056128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4280343 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81588562525184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2188.966818001 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2846984186368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10157908 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2842688910422 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24253954 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44095350 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19841396 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44085191 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19841396 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19841394 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚375758354068214 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2842348349036 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _step‚200^¬û}ï&» éM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4280346 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24243797 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚377957085811200 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚84452236573184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2863579271680 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5877565 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44095350 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚372915665157792 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377957444281856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚377957444281856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚375110460095488 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2863674048000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚375110443056128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4280343 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81588562525184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2188.966818001 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2846984186368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10157908 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2842688910422 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24253954 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44095350 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19841396 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44085191 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19841396 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19841394 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚375758354068214 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2842348349036 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _step‚200 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 % _timestamp‚1.7845090153910942e+09ÈmG6ž2¼ - _wandbruntime‚2188  _runtime‚2188'%‡³½ þ $loss/total_loss‚8.90920001624654 epoch‚6 trainer/global_step‚2449 0loss/FRt1/assignment_loss‚1.8834940282007058 2loss/SRqqt1/assignment_loss‚0.5661524073724119 2loss/SRqqt2/assignment_loss‚0.5687238668154693 _runtime‚2188.967313132 /loss/FBt2/detection_loss‚0.8301654736009928 $ _timestamp‚1.784509015392063e+09 _step‚201 /loss/FRt2/assignment_loss‚0.961828811589731 1loss/FBt1/assignment_loss‚0.02044789963546819 1loss/SRqqt2/detection_loss‚0.7045825389883389 /loss/FBt1/detection_loss‚0.5323030602172479 1loss/FBt2/assignment_loss‚0.02141733657891729 /loss/FRt1/detection_loss‚0.5051921427417861 /loss/FRt2/detection_loss‚1.5284340724748466 1loss/SRqqt1/detection_loss‚0.786458378030623680Þf ¾ ƒ 1loss/FBt2/assignment_loss‚0.02141733657891729 /loss/FRt1/detection_loss‚0.5051921427417861 /loss/FRt2/detection_loss‚1.5284340724748466 1loss/SRqqt1/detection_loss‚0.7864583780306236 $loss/total_loss‚8.90920001624654 epoch‚6 trainer/global_step‚2449 0loss/FRt1/assignment_loss‚1.8834940282007058 2loss/SRqqt1/assignment_loss‚0.5661524073724119 2loss/SRqqt2/assignment_loss‚0.5687238668154693 _runtime‚2188.967313132 /loss/FBt2/detection_loss‚0.8301654736009928 $ _timestamp‚1.784509015392063e+09 _step‚201 /loss/FRt2/assignment_loss‚0.961828811589731 1loss/FBt1/assignment_loss‚0.02044789963546819 1loss/SRqqt2/detection_loss‚0.7045825389883389 /loss/FBt1/detection_loss‚0.5323030602172479Éž5‚¸Y¿ ‚(:Î ØäõÒ¥”üË# disk.dm-0.out‚18539.59912109375% disk.nvme0n1.out‚14641.5947265625 disk.nvme1c1n1.in‚0 network.sent‚ 3.733041e+06 disk.md0.out‚ 69.30859375 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme1n1.out‚7702.8974609375 proc.memory.rssMB‚ 2987.09375* proc.memory.percent‚18.231773376464844 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 739.4921875 cpu‚25.629496849165232 network.recv‚590903" disk.dm-1.out‚3657.19091796875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚14641.5947265625 disk.nvme0n1.in‚ 0.00390625& disk.nvme1c1n1.out‚7702.8974609375$ memory_percent‚95.48649787902832 proc.cpu.threads‚17) disk./.usagePercent‚76.14443179082258% disk./.usageGB‚1169.0073890686035¼ña“À ‚(:ˆ ØäõÒƒ©ÄÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚275.082' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€‡½VÁ ‚(:Ë çäõÒ„˜±Ë proc.memory.rssMB‚ 2987.09375 proc.cpu.threads‚17 network.sent‚ 3.782409e+06 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 network.recv‚595204% disk./.usageGB‚1169.0067291259766& disk.nvme0n1.out‚14656.69189453125 disk.nvme1n1.in‚0 disk.md0.out‚ 69.69921875 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0$ memory_percent‚95.48673629760742) disk./.usagePercent‚76.14438880481853% disk.nvme1c1n1.out‚7705.341796875# disk.nvme1n1.out‚7705.341796875& proc.memory.availableMB‚ 739.453125* proc.memory.percent‚18.231773376464844 cpu‚25.628523886539217! disk.dm-0.out‚18556.205078125 disk.nvme0n1.in‚ 0.00390625 disk.md0.in‚0" disk.dm-1.out‚3657.27685546875( disk.nvme0c0n1.out‚14656.69189453125€£àž ‚(:“ çäõÒ¡…Ì gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚282.526' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.17533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øºšTà ‚(:É öäõÒ‚¬àË% disk./.usageGB‚1169.0067100524902 disk.dm-1.in‚0$ disk.nvme0n1.out‚14657.826171875$ disk.nvme1n1.out‚7706.2353515625) disk./.usagePercent‚76.14438756244849! disk.dm-0.out‚18556.853515625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚14657.826171875& disk.nvme1c1n1.out‚7706.2353515625 cpu‚25.627541143701905 network.sent‚ 3.788944e+06 disk.md0.out‚ 70.29296875! disk.dm-1.out‚3657.3955078125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ memory_percent‚95.48678398132324 network.recv‚597751 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625' proc.memory.availableMB‚ 739.4453125 proc.memory.rssMB‚ 2987.09375* proc.memory.percent‚18.231773376464844 proc.cpu.threads‚17ÆŒ/à“Ä ‚(:ˆ öäõÒàþ©Ì gpu.0.gpu‚54.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.863' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.621 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VrØ2Å - _wandbruntime‚2230  _runtime‚2230Poé ÆÆ À & lr-AdamW/pg1‚0.0005972641333877234 & lr-AdamW/pg2‚0.0005972641333877234 trainer/global_step‚2499 % _timestamp‚1.7845090573098044e+09 _runtime‚ 2230.88543224 _step‚202œ‡GËÇ Å _step‚202 & lr-AdamW/pg1‚0.0005972641333877234 & lr-AdamW/pg2‚0.0005972641333877234 trainer/global_step‚2499 % _timestamp‚1.7845090573098044e+09 _runtime‚ 2230.88543224ÊÔ÷>¦2È - _wandbruntime‚2230  _runtime‚2230ªÛ»E™ É ÛO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20223253 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚382375910700544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚385276995419136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4365309 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5990389 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2918368936960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚203 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚382375910700544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2897063735466 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20223253 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20223251 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10355985 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24728936 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚385277353882624 % _timestamp‚1.7845090573107717e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24718793 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44952189 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86132872062464 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44942044 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2901443182080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380139767863920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4365312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24728936 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2896723174192 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚383036831599386 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86132746083840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83214395759104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2901101757952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚385276995419136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2918476303360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44942044 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2901443182080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚380139767863920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4365312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24728936 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2896723174192 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚383036831599386 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86132746083840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83214395759104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2901101757952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚385276995419136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2918476303360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚385276995419136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4365309 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5990389 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2918368936960 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚203 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚382375910700544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2897063735466 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20223253 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20223251 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10355985 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24728936 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚385277353882624 % _timestamp‚1.7845090573107717e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24718793 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799Ëp‹Çþ2Ë - _wandbruntime‚2231  _runtime‚2231˜‰è&Ì âM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚383185370655222 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44959493 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44969652 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10360008 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44969652 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 % _timestamp‚1.7845090579258516e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10359714 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4367046 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24728449 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86167162431488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83247557416960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚382524168163328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5992962 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380287181492032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚382524168163328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2902553288192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24738606 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2898172123830 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _runtime‚ 2231.501809 _step‚204 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚382524185202688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20231046 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83247576029184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20231046 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4367043 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86167038557184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383185013054476 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚385426738490880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5992671 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2902211856896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380287198531392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2902553288192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚382524185202688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24728449 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2919586402304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24738606 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2902211856896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2897831562444 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581§Jò•Í çM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚382524185202688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20231046 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83247576029184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20231046 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4367043 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86167038557184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383185013054476 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚385426738490880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5992671 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2902211856896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚380287198531392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2902553288192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚382524185202688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24728449 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2919586402304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24738606 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2902211856896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2897831562444 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pooãÉGXl.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚383185370655222 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44959493 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44969652 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10360008 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44969652 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 % _timestamp‚1.7845090579258516e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10359714 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4367046 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24728449 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86167162431488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83247557416960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚382524168163328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5992962 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380287181492032 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚382524168163328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2902553288192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24738606 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2898172123830 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _runtime‚ 2231.501809 _step‚204 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488Ììl!2Î - _wandbruntime‚2231  _runtime‚2231òœ.Ï  0loss/SRqqt1/detection_loss‚0.781395463426697 trainer/global_step‚2499 epoch‚6 4loss/FBt1/assignment_loss‚0.00022469017494186616 _step‚205 0loss/FRt2/assignment_loss‚0.9240751484688142 2loss/SRqqt1/assignment_loss‚0.5786107059883795 .loss/FRt1/detection_loss‚0.508878463964123 /loss/FRt2/detection_loss‚1.7149035331074616 % _timestamp‚1.7845090579270387e+09 0loss/FRt1/assignment_loss‚1.9177598357200623 1loss/SRqqt2/assignment_loss‚0.596039780916893 1loss/FBt2/assignment_loss‚0.04368837217587718 1loss/SRqqt2/detection_loss‚0.7212138013594807 /loss/FBt1/detection_loss‚0.7083189217828476 /loss/FBt2/detection_loss‚0.7693492574617267 %loss/total_loss‚9.264457974547305 _runtime‚2231.502372531y¯] Ð † epoch‚6 4loss/FBt1/assignment_loss‚0.00022469017494186616 _step‚205 0loss/FRt2/assignment_loss‚0.9240751484688142 2loss/SRqqt1/assignment_loss‚0.5786107059883795 .loss/FRt1/detection_loss‚0.508878463964123 /loss/FRt2/detection_loss‚1.7149035331074616 % _timestamp‚1.7845090579270387e+09 0loss/FRt1/assignment_loss‚1.9177598357200623 1loss/SRqqt2/assignment_loss‚0.596039780916893 1loss/FBt2/assignment_loss‚0.04368837217587718 1loss/SRqqt2/detection_loss‚0.7212138013594807 /loss/FBt1/detection_loss‚0.7083189217828476 /loss/FBt2/detection_loss‚0.7693492574617267 %loss/total_loss‚9.264457974547305 _runtime‚2231.502372531 0loss/SRqqt1/detection_loss‚0.781395463426697 trainer/global_step‚2499Íàï¤WÑ ‚(:Ì …åõÒ°ëË( disk.nvme0c0n1.out‚14659.83251953125& disk.nvme0n1.out‚14659.83251953125# memory_percent‚95.4871654510498) disk./.usagePercent‚76.14439104108463& disk.nvme1c1n1.out‚7706.9619140625 disk.md0.out‚ 70.6953125 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625* proc.memory.percent‚18.231773376464844 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚7706.9619140625 disk.md0.in‚0 proc.memory.rssMB‚ 2987.09375 disk.nvme1n1.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚3657.4619140625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625' proc.memory.availableMB‚ 739.3828125 cpu‚25.626356132649178 network.sent‚ 3.803963e+06 network.recv‚601041$ disk./.usageGB‚1169.006763458252# disk.dm-0.out‚18558.65283203125 disk.rbd0.in‚0Ž²…jžÒ ‚(:“ …åõÒ¾§³Ì gpu.0.gpu‚63.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.069' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.35633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z[RÓ ‚(:Ç ”åõÒó¬ÂË network.recv‚604794( disk./.usagePercent‚76.1444367603028 disk.rbd0.in‚0( disk.nvme0c0n1.out‚14661.23583984375 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 7707.8671875 disk.md0.in‚0* proc.memory.percent‚18.231773376464844 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.dm-1.in‚0( proc.memory.availableMB‚ 739.20703125 proc.cpu.threads‚17% disk./.usageGB‚1169.0074653625488 disk.nvme1c1n1.in‚0# disk.nvme1c1n1.out‚ 7707.8671875 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚3657.65380859375 proc.memory.rssMB‚ 2987.09375 cpu‚25.625306716292606 network.sent‚ 3.855503e+06& disk.nvme0n1.out‚14661.23583984375 disk.md0.out‚ 71.140625" disk.dm-0.out‚18559.8115234375$ memory_percent‚95.48823833465576Æû\Ô ‚(:’ ”åõÒÄË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚123.98' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.32666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôM2QÕ ‚(:Æ £åõÒ€ŠÊË$ memory_percent‚95.48687934875488' proc.memory.availableMB‚ 739.4296875& disk.nvme0n1.out‚14664.56103515625 network.sent‚ 3.86044e+06 network.recv‚608425 disk.nvme1n1.in‚0 disk.md0.in‚0* proc.memory.percent‚18.231773376464844 cpu‚25.624262150826723 proc.cpu.threads‚17) disk./.usagePercent‚76.14442036101804 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.md0.out‚ 71.453125 proc.memory.rssMB‚ 2987.09375 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚18563.23828125 disk.dm-1.in‚0" disk.dm-1.out‚3657.69677734375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚14664.56103515625 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 7708.69921875% disk./.usageGB‚1169.0072135925293 disk.nvme0n1.in‚ 0.00390625" disk.nvme1n1.out‚ 7708.69921875îÆ‚žÖ ‚(:“ £åõÒÒ¼‹Ì gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚282.007' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.00233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SQY×2× - _wandbruntime‚2272  _runtime‚2272Wà”/ÇØ Á & lr-AdamW/pg2‚0.0005964419412045149 trainer/global_step‚2549 % _timestamp‚1.7845090993250585e+09 _runtime‚2272.902029511 _step‚206 & lr-AdamW/pg1‚0.0005964419412045149ÿÿ`ÌÙ Æ % _timestamp‚1.7845090993250585e+09 _runtime‚2272.902029511 _step‚206 & lr-AdamW/pg1‚0.0005964419412045149 & lr-AdamW/pg2‚0.0005964419412045149 trainer/global_step‚2549ÎÑ¥©2Ú - _wandbruntime‚2272  _runtime‚22722;rà'Û ÚO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6106317 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25203445 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25213588 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4452009 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84873390650880 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2952206387600 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚389789635807744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392746648091648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25213588 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2956670859776 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392746289628160 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87847761715200 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87847629445120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2974352452096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974238794240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2957012283904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20612903 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390463490585760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387511284198160 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2956670859776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45826491 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4452012 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2957012283904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45816346 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 $ _timestamp‚1.784509099325686e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _runtime‚2272.902099612 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚392746648091648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10558329 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10558034 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _step‚207 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚389789618768384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389789635807744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84873409263104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534ÂdTÙå'Ü ßO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6106317 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25203445 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25213588 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4452009 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84873390650880 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2952206387600 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚389789635807744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392746648091648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25213588 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2956670859776 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392746289628160 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87847761715200 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87847629445120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2974352452096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974238794240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2957012283904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20612903 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390463490585760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387511284198160 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2956670859776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45826491 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4452012 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2957012283904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45816346 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 $ _timestamp‚1.784509099325686e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _runtime‚2272.902099612 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚392746648091648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10558329 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10558034 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _step‚207 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚389789618768384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389789635807744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84873409263104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Ïúo¡2Ý - _wandbruntime‚2273  _runtime‚2273ìÔ7Ó$Þ åM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87881923646976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87882055909888 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25223258 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25213101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10562301 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2975466376704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocationsŠ vÞÇ .freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20620694 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2957780958720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390612387242230 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚387658714865632 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚392895674229248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20620696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389937910309888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚387658731904992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2953655337238 _runtime‚2273.517371509 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45843954 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2953314775852 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25223258 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2957780958720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2958122390016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚2549 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10562012 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390612029641484 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845090999413586e+09v¸AÄð&ß êM ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2953314775852 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25223258 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2957780958720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2958122390016 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚2549 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10562012 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390612029641484 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845090999413586e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87881923646976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87882055909888 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25223258 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25213101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10562301 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2975466376704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20620694 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2957780958720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390612387242230 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚387658714865632 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚392895674229248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20620696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389937910309888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚387658731904992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2953655337238 _runtime‚2273.517371509 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45843954 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688оùâ­2à - _wandbruntime‚2273  _runtime‚2273ø˜S¢ á ƒ epoch‚6 % _timestamp‚1.7845090999426239e+09 2loss/SRqqt2/assignment_loss‚0.6589167399367363 4loss/FBt2/assignment_loss‚2.8923524757432417e-05 /loss/FBt1/detection_loss‚0.4227437741074321 /loss/FBt2/detection_loss‚0.7641930191582719 trainer/global_step‚2549 2loss/FBt1/assignment_loss‚0.052874034222831996 /loss/FRt1/detection_loss‚0.5287041639911169 1loss/SRqqt1/detection_loss‚0.6856049447600281 0loss/SRqqt2/detection_loss‚0.754793189877874 %loss/total_loss‚8.892343522496974 _step‚209 0loss/FRt1/assignment_loss‚1.9165781414170886 0loss/FRt2/assignment_loss‚0.8955809533731812 2loss/SRqqt1/assignment_loss‚0.5633861811884674 /loss/FRt2/detection_loss‚1.6489394569391882 _runtime‚ 2273.51793549³ó.â ˆ /loss/FBt1/detection_loss‚0.4227437741074321 /loss/FBt2/detection_loss‚0.7641930191582719 trainer/global_step‚2549 2loss/FBt1/assignment_loss‚0.052874034222831996 /loss/FRt1/detection_loss‚0.5287041639911169 1loss/SRqqt1/detection_loss‚0.6856049447600281 0loss/SRqqt2/detection_loss‚0.754793189877874 %loss/total_loss‚8.892343522496974 _step‚209 0loss/FRt1/assignment_loss‚1.9165781414170886 0loss/FRt2/assignment_loss‚0.8955809533731812 2loss/SRqqt1/assignment_loss‚0.5633861811884674 /loss/FRt2/detection_loss‚1.6489394569391882 _runtime‚ 2273.51793549 epoch‚6 % _timestamp‚1.7845090999426239e+09 2loss/SRqqt2/assignment_loss‚0.6589167399367363 4loss/FBt2/assignment_loss‚2.8923524757432417e-05Ñã),jUã ‚(:Ê ²åõÒùåÂË& proc.memory.availableMB‚ 739.390625* proc.memory.percent‚18.231773376464844) disk./.usagePercent‚76.14443551793275% disk./.usageGB‚1169.0074462890625 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.out‚ 3657.828125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚14666.1982421875 disk.nvme0n1.in‚ 0.00390625 network.recv‚611846# disk.dm-0.out‚18564.56201171875% disk.nvme0n1.out‚14666.1982421875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚7709.42529296875 cpu‚25.623342893557858 proc.cpu.threads‚17 network.sent‚ 3.876205e+06 disk.dm-1.in‚0$ memory_percent‚95.48711776733398 proc.memory.rssMB‚ 2987.09375 disk.md0.out‚ 71.87890625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚7709.42529296875^,Yöœä ‚(:‘ ²åõÒä ¯Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.875' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.95833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 NªÙSå ‚(:È ÁåõÒŽäæË! disk.dm-0.out‚18567.126953125 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚14668.99462890625& proc.memory.availableMB‚ 739.390625 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚14668.99462890625 proc.memory.rssMB‚ 2987.09375 disk.dm-1.out‚ 3657.890625% disk.nvme1n1.out‚7710.20361328125 disk.md0.in‚0 disk.dm-1.in‚0$ memory_percent‚95.48711776733398 proc.cpu.threads‚17 network.recv‚616303) disk./.usagePercent‚76.14443576640676 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚7710.20361328125 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚18.231773376464844 cpu‚25.62231954235641 network.sent‚ 3.927561e+06% disk./.usageGB‚1169.0074501037598 disk.nvme1n1.in‚0 disk.md0.out‚ 72.3203125 ÞÕažæ ‚(:“ ÁåõÒ¨«¯Ì gpu.0.gpu‚61.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚145.238' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚48.41266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉUç ‚(:Ê ÐåõÒŠåßË# disk.dm-0.out‚18568.33154296875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚14670.6962890625 disk.nvme1c1n1.in‚0* proc.memory.percent‚18.231773376464844# disk.nvme1n1.out‚7711.138671875" disk.dm-1.out‚3658.12548828125' proc.memory.availableMB‚ 739.3828125 cpu‚25.621317080923145 proc.cpu.threads‚17 network.sent‚ 3.932352e+06) disk./.usagePercent‚76.14444669926328 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme1c1n1.out‚7711.138671875# memory_percent‚95.4871654510498 network.recv‚619784% disk./.usageGB‚1169.0076179504395 disk.nvme1n1.in‚0 disk.md0.out‚ 72.8828125 disk.dm-0.in‚ 0.00390625' disk.nvme0c0n1.out‚14670.6962890625 proc.memory.rssMB‚ 2987.09375w˜Êžè ‚(:“ ÐåõÒÒ€§Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚289.907' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.63566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä/X2é - _wandbruntime‚2315  _runtime‚2315;4-ªÇê Á % _timestamp‚1.7845091417203257e+09 _runtime‚2315.299036894 _step‚210 & lr-AdamW/pg1‚0.0005956011744236154 & lr-AdamW/pg2‚0.0005956011744236154 trainer/global_step‚2599f¯§{Ìë Æ & lr-AdamW/pg2‚0.0005956011744236154 trainer/global_step‚2599 % _timestamp‚1.7845091417203257e+09 _runtime‚2315.299036894 _step‚210 & lr-AdamW/pg1‚0.0005956011744236154Òx¶;‡2ì - _wandbruntime‚2315  _runtime‚2315v|v²Ä í ÚO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3007689601008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6221545 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚2315.299099555 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10759969 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4538712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845091417210069e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86532404154880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚397890864773402 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚400215942300672 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89562489242624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25688097 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397203360914944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46690648 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25698240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚397890507172768 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10760257 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚400215583837184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚400215583837184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46700793 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21002553 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397203343875584 SGDeviceStatsMonitor.on_traæ6$in_batch_start/inactive_split.small_pool.freed‚6221260 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89562602638336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚2599 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3012581385728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3012239961600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚400215942300672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3030179871232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚394882817571760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397203343875584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86532422767104–Çãå'î ßO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚400215942300672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3030179871232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚394882817571760 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397203343875584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86532422767104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3007689601008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6221545 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚2315.299099555 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10759969 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4538712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845091417210069e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86532404154880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚397890864773402 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚400215942300672 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89562489242624 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25688097 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚397203360914944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46690648 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25698240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚397890507172768 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10760257 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚400215583837184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚400215583837184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46700793 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21002553 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397203343875584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6221260 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89562602638336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚2599 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3012581385728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3012239961600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180Ó Á¢Í2ï - _wandbruntime‚2315  _runtime‚2315ÏÖSºë&ð åM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3031303395328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21010344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚2315.908329838 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10763959 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395030248239232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21010346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397351618377728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397351618377728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚212 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6223516 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400365326908928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400365326908928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86565603037184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3031196036096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21010346 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6223807 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86565584424960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395030265278592 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3013350060544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25707910 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46718256 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚400364968438272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4540443 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25697753 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3008797989260 % _timestamp‚1.7845091423326263e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46718256 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46708097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚2599 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184¾Í1Nð&ñ êM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4540443 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25697753 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3008797989260 % _timestamp‚1.7845091423326263e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46718256 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46708097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚2599 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3031303395328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21010344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚2315.908329838 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10763959 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395030248239232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21010346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397351618377728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚397351618377728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚212 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6223516 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400365326908928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400365326908928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚86565603037184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3031196036096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21010346 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6223807 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86565584424960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395030265278592 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3013350060544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25707910 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46718256 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚400364968438272ÔóéQ+2ò - _wandbruntime‚2315  _runtime‚2315ÜÜÁãó ƒ /loss/FRt1/assignment_loss‚2.005779766632719 0loss/FRt2/assignment_loss‚0.8706242521603902 1loss/SRqqt1/detection_loss‚0.7469714655197764 trainer/global_step‚2599 /loss/FRt1/detection_loss‚0.5117078040025336 /loss/FRt2/detection_loss‚1.6648318816861714 1loss/SRqqt2/detection_loss‚0.7396641140085178 0loss/FBt1/detection_loss‚0.39322753241670816 %loss/total_loss‚8.669095005355743 $ _timestamp‚1.784509142334659e+09 _runtime‚2315.909952921 _step‚213 2loss/SRqqt2/assignment_loss‚0.6216734568672606 2loss/FBt1/assignment_loss‚0.007296169991305381 3loss/FBt2/assignment_loss‚0.00105180151sŒ´Åz88769184 epoch‚6 2loss/SRqqt1/assignment_loss‚0.5474173849267812 /loss/FBt2/detection_loss‚0.55884937562470178ö0ýô ˆ 2loss/SRqqt2/assignment_loss‚0.6216734568672606 2loss/FBt1/assignment_loss‚0.007296169991305381 3loss/FBt2/assignment_loss‚0.0010518015188769184 epoch‚6 2loss/SRqqt1/assignment_loss‚0.5474173849267812 /loss/FBt2/detection_loss‚0.5588493756247017 /loss/FRt1/assignment_loss‚2.005779766632719 0loss/FRt2/assignment_loss‚0.8706242521603902 1loss/SRqqt1/detection_loss‚0.7469714655197764 trainer/global_step‚2599 /loss/FRt1/detection_loss‚0.5117078040025336 /loss/FRt2/detection_loss‚1.6648318816861714 1loss/SRqqt2/detection_loss‚0.7396641140085178 0loss/FBt1/detection_loss‚0.39322753241670816 %loss/total_loss‚8.669095005355743 $ _timestamp‚1.784509142334659e+09 _runtime‚2315.909952921 _step‚213Õ.cÓÈYõ ‚(:Î ßåõÒÖ¡ÅË" disk.dm-1.out‚3658.21923828125 disk.rbd0.out‚0 network.sent‚ 3.935274e+06$ disk./.usageGB‚1169.007656097412 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚7711.9345703125 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0( proc.memory.availableMB‚ 492.90234375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚14672.9541015625 proc.memory.rssMB‚ 2987.09375* proc.memory.percent‚18.231773376464844 proc.cpu.threads‚17% disk.nvme0n1.out‚14672.9541015625 disk.md0.out‚ 73.30078125$ memory_percent‚96.99156284332275 cpu‚25.620437771973272 network.recv‚623569) disk./.usagePercent‚76.14444918400339 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚7711.9345703125# disk.dm-0.out‚18570.38720703125Ugü¹’ö ‚(:‡ ßåõÒÏ£´Ì gpu.0.gpu‚33.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚264.873' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.291 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`; V÷ ‚(:Ë îåõÒ½ðÓË! disk.dm-1.out‚3658.3193359375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚14673.9619140625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚7712.65185546875 disk.md0.out‚ 73.765625# disk.dm-0.out‚18571.04345703125' proc.memory.availableMB‚ 493.0234375* proc.memory.percent‚18.231773376464844 network.recv‚627653% disk./.usageGB‚1169.0075607299805% disk.nvme0n1.out‚14673.9619140625 disk.md0.in‚0$ memory_percent‚96.99082374572754 proc.memory.rssMB‚ 2987.09375 proc.cpu.threads‚17' disk.nvme1c1n1.out‚7712.65185546875 disk.dm-0.in‚ 0.00390625 cpu‚25.61924406888493 network.sent‚ 3.999509e+06( disk./.usagePercent‚76.1444429721531 disk.dm-1.in‚0”Óœø ‚(:‘ îåõÒÔã—Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.874' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.29133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘WWYù ‚(:Î ýåõÒáâÌË$ memory_percent‚96.99082374572754 proc.memory.rssMB‚ 2987.09375 cpu‚25.618280135254672 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚7713.47314453125 disk.nvme1n1.in‚0* proc.memory.percent‚18.231773376464844 network.sent‚ 4.004303e+06) disk./.usagePercent‚76.14444048741298% disk./.usageGB‚1169.0075225830078! disk.dm-1.out‚3658.3583984375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme0n1.out‚14675.7373046875% disk.nvme1n1.out‚7713.47314453125 network.recv‚631346 disk.dm-1.in‚0' disk.nvme0c0n1.out‚14675.7373046875 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 74.3671875# disk.dm-0.out‚18572.34130859375' proc.memory.availableMB‚ 493.0234375 disk.dm-0.in‚ 0.00390625ƒMþžú ‚(:“ ýåõÒ¹Ñ”Ì gpu.0.gpu‚91.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚287.575' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.85833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á}»W2û - _wandbruntime‚2357  _runtime‚2357a¤-PÇü Á & lr-AdamW/pg1‚0.0005947418870459043 & lr-AdamW/pg2‚0.0005947418870459043 trainer/global_step‚2649 % _timestamp‚1.7845091838810782e+09 _runtime‚2357.458036646 _step‚214Óñ®.Ìý Æ & lr-AdamW/pg1‚0.0005947418870459043 & lr-AdamW/pg2‚0.0005947418870459043 trainer/global_step‚2649 % _timestamp‚1.7845091838810782e+09 _runtime‚2357.458036646 _step‚214Ö}äˆ2þ - _wandbruntime‚2357  _runtime‚2357³†«áá'ÿ ÛO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407685236509696 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26172749 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3085943191552 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3063513375690 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47575095 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚404617068982784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚404617068982784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21392203 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21392201 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407684878046208 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3067809063424 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26182892 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚402254367984720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚402254350945360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚215 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26172749 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3063172814416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3068150487552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88191417658880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4625412 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6337219 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6336940 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88191436271104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚404617086022144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚2649 % _timestamp‚1.7845091838816798e+09 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4625412 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6337219 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6336940 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88191436271104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚404617086022144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚2649 % _timestamp‚1.7845091838816798e+09 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407684878046208 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3067809063424 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26182892 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚402254367984720 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚402254350945360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚215 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26172749 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3063172814416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3068150487552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88191417658880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0×ͺu2 - _wandbruntime‚2358  _runtime‚2358g{D7D‚ äM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26192562 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6339250 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47592558 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88224597928960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21399994 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3064281202668 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26182405 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26182405 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6339527 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4627146 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3068919162368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4627143 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405466062815500 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 trainer/global_step‚2649 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21399994 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10966673 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407834262647296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47582399 $ _timestamp‚1.784509184799215e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47582399 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3069260593664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91311777197568 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88224616541184 _step‚216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407834621117952 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚407834621117952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚404765360524288 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21399996 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚402401781612832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.Ì*¦ all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3069260593664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10966393 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3087059588608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47592558 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26192562 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896ôPå×ï&ƒ éM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21399994 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10966673 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407834262647296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47582399 $ _timestamp‚1.784509184799215e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47582399 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3069260593664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91311777197568 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88224616541184 _step‚216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407834621117952 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚407834621117952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚404765360524288 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21399996 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚402401781612832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3069260593664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10966393 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3087059588608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47592558 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26192562 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26192562 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6339250 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47592558 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88224597928960 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21399994 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3064281202668 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26182405 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26182405 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6339527 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4627146 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3068919162368 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4627143 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405466062815500 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 trainer/global_step‚2649 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303ØQ#ÞÍ2„ - _wandbruntime‚2358  _runtime‚23582t … ƒ /loss/FRt1/detection_loss‚0.5399053259359017 0loss/SRqqt2/detection_loss‚0.679573459439139 trainer/global_step‚2649 0loss/FRt2/assignment_loss‚0.9518365411414313 2loss/SRqqt2/assignment_loss‚0.5673006518899872 2loss/FBt1/assignment_loss‚0.034372265760856766 1loss/SRqqt1/detection_loss‚0.8957146179442779 % _timestamp‚1.7845091848001845e+09 _runtime‚2358.375517079 2loss/SRqqt1/assignment_loss‚0.5575733024703867 2loss/FBt2/assignment_loss‚0.001987042024750959 /loss/FRt2/detection_loss‚1.5272508300668033 0loss/FBt1/detection_loss‚0.45824664898423684 /loss/FBt2/detection_loss‚0.7904262960317635 %loss/total_loss‚8.949995728088096 _step‚217 epoch‚6 0loss/FRt1/assignment_loss‚1.9458087463985607°_Ä9† ˆ 2loss/SRqqt2/assignment_loss‚0.5673006518899872 2loss/FBt1/assignment_loss‚0.034372265760856766 1loss/SRqqt1/detection_loss‚0.8957146179442779 % _timestamp‚1.7845091848001845e+09 _runtime‚2358.375517079 2loss/SRqqt1/assignment_loss‚0.5575733024703867 2loss/FBt2/assignment_loss‚0.001987042024750959 /loss/FRt2/detection_loss‚1.5272508300668033 0loss/FBt1/detection_loss‚0.45824664898423684 /loss/FBt2/detection_loss‚0.7904262960317635 %loss/total_loss‚8.949995728088096 _step‚217 epoch‚6 0loss/FRt1/assignment_loss‚1.9458087463985607 /loss/FRt1/detection_loss‚0.5399053259359017 0loss/SRqqt2/detection_loss‚0.679573459439139 trainer/global_step‚2649 0loss/FRt2/assignment_loss‚0.9518365411414313Ù¦Cq\‡ ‚(:Ñ ŒæõÒ„ÀË) disk./.usagePercent‚76.14444421452316 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625 disk.dm-1.in‚0( proc.memory.availableMB‚ 492.98828125 proc.memory.rssMB‚ 2987.09375* proc.memory.percent‚18.231773376464844 cpu‚25.61743408811067 network.sent‚ 4.02011e+06& disk.nvme0n1.out‚14676.67919921875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚7714.15771484375 disk.md0.out‚ 74.77734375 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚18572.92333984375 network.recv‚634646 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚14676.67919921875' disk.nvme1c1n1.out‚7714.15771484375 disk.md0.in‚0" disk.dm-1.out‚3658.52685546875$ memory_percent‚96.99103832244873% disk./.usageGB‚1169.0075798034668 proc.cpu.threads‚17Cipžˆ ‚(:“ ŒæõÒÚïüË gpu.0.gpu‚67.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚204.073' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚68.02433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00„CW‰ ‚(:Ì ›æõÒÝéàË" disk.dm-0.out‚18575.2294921875$ memory_percent‚96.99032306671143 cpu‚25.61659066693269 disk.md0.in‚0( proc.memory.availableMB‚ 493.10546875 proc.cpu.threads‚17 network.sent‚ 4.071439e+06 network.recv‚638903) disk./.usagePercent‚76.14444843858136 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚14679.04443359375% disk./.usageGB‚1169.0076446533203 disk.nvme1n1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚14679.04443359375 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚7714.681640625 proc.memory.rssMB‚ 2987.09375* proc.memory.percent‚18.231773376464844" disk.dm-1.out‚3658.56982421875 disk.rbd0.out‚0# disk.nvme1n1.out‚7714.681640625 disk.md0.out‚ 75.0234375¤¾$&žŠ ‚(:“ ›æõÒéí£Ì gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚269.561' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.85366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥Æÿ2‹ - _wandbruntime‚2399  _runtime‚2399hú„ÄŒ ¾ % lr-AdamW/pg1‚0.000593864134261804 % lr-AdamW/pg2‚0.000593864134261804 trainer/global_step‚2699 $ _timestamp‚1.784509226338408e+09 _runtime‚2399.913993151 _step‚218fz —É Ã % lr-AdamW/pg1‚0.000593864134261804 % lr-AdamW/pg2‚0.000593864134261804 trainer/global_step‚2699 $ _timestamp‚1.784509226338408e+09 _runtime‚2399.913993151 _step‚218Ú9_& 2Ž - _wandbruntime‚2399  _runtime‚23992=;ß' ÙO _step‚219 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6453001 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚2699 _runtime‚2399.914761602 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21781851 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3123378165248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3118996589098 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26657401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415154530718720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚412030794089984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚412030811129344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3123378165248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89850431162880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415154172255232 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92992334179840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21781853 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48449397 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚412744897947418 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4712109 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11165113 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48439252 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26667544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6452718 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚409625901358320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚412030811129344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21781851 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3118656027824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3123719589376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3141884404736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26667544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415154172255232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚409625884318960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232…ªpÒú ÞO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚415154172255232 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚409625884318960 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚219 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6453001 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚2699 _runtime‚2399.914761602 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21781851 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3123378165248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3118996589098 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26657401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚14Cø^ê52 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚415154530718720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚412030794089984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚412030811129344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3123378165248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89850431162880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚415154172255232 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚92992334179840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21781853 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48449397 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚412744897947418 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4712109 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11165113 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48439252 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26667544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6452718 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚409625901358320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚412030811129344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21781851 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3118656027824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3123719589376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3141884404736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26667544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224Û+²¿}Q‘ ‚(:Æ ªæõÒÚŒ´Ë disk.md0.in‚0" disk.dm-0.out‚18575.8349609375$ memory_percent‚96.99056148529053 proc.memory.rssMB‚ 2987.09375 network.sent‚ 4.087246e+06 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.out‚ 7716.28125 disk.dm-1.in‚0* proc.memory.percent‚18.231773376464844 cpu‚25.615667651925204( disk.nvme0c0n1.out‚14680.91064453125 proc.cpu.threads‚17 network.recv‚642183) disk./.usagePercent‚76.14444098436101 disk.rbd0.out‚0& disk.nvme0n1.out‚14680.91064453125 disk.md0.out‚ 76.37890625 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚3658.65087890625( proc.memory.availableMB‚ 493.06640625% disk./.usageGB‚1169.0075302124023 disk.nvme1c1n1.in‚0! disk.nvme1c1n1.out‚ 7716.28125 disk.nvme1n1.in‚0:î1ž’ ‚(:“ ªæõÒôãúË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚286.307' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.43566666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò;ÉL2“ - _wandbruntime‚2400  _runtime‚2400ðÖ*·é&” ãM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48466860 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚412179068592128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚409773314986432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3119764416076 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21789646 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26667057 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415303915326976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48456701 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21789644 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26677214 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11169091 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93026627301376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4713843 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89883611432960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21789644 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412893079402508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _step‚220 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3120104977462 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚412179085631488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21789646 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409773332025792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚412179085631488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4713846 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415303556856320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3124829695488 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11168822 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3142997256192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26677214 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48466860 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269šËeéî&• èM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21789644 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412893079402508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _step‚220 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3120104977462 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚412179085631488 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21789646 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409773332025792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚412179085631488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4713846 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚415303556856320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3124829695488 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11168822 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3142997256192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26677214 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48466860 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48466860 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚412179068592128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚409773314986432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3119764416076 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21789646 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26667057 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚415303915326976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48456701 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21789644 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26677214 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11169091 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93026627301376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4713843 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89883611432960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452Ün¢b“2– - _wandbruntime‚2400  _runtime‚2400øb€— ‚ /loss/FRt1/detection_loss‚0.5574291977131833 1loss/SRqqt2/detection_loss‚0.7218400640014903 epoch‚6 _runtime‚2400.528220071 _step‚221 /loss/FRt1/assignment_loss‚1.918174067591006 1loss/SRqqt1/detection_loss‚0.7176892919569625 0loss/FBt1/detection_loss‚0.44883147500877385 $ _timestamp‚1.784509226952842e+09 0loss/FRt2/assignment_loss‚0.8785191301591785 2loss/SRqqt2/assignment_loss‚0.6057767271787796 2loss/FBt2/assignment_loss‚0.010107310262598833 %loss/total_loss‚8.804575333366987 2loss/SRqqt1/assignment_loss‚0.6481086160101122 2loss/FBt1/assignment_loss‚0.023158912125209763 /loss/FRt2/detection_loss‚1.6231236399044884 /loss/FBt2/detection_loss‚0.6518169014552047 trainer/global_step‚2699ê0õ ˜ ‡ /loss/FRt2/detection_loss‚1.6231236399044884 /loss/FBt2/detection_loss‚0.6518169014552047 trainer/global_step‚2699 /loss/FRt1/detection_loss‚0.5574291977131833 1loss/SRqqt2/detection_loss‚0.7218400640014903 epoch‚6 _runtime‚2400.528220071 _step‚221 /loss/FRt1/assignment_loss‚1.918174067591006 1loss/SRqqt1/detection_loss‚0.7176892919569625 0loss/FBt1/detection_loss‚0.44883147500877385 $ _timestamp‚1.784509226952842e+09 0loss/FRt2/assignment_loss‚0.8785191301591785 2loss/SRqqt2/assignment_loss‚0.6057767271787796 2loss/FBt2/assignment_loss‚0.010107310262598833 %loss/total_loss‚8.804575333366987 2loss/SRqqt1/assignment_loss‚0.6481086160101122 2loss/FBt1/assignment_loss‚0.023158912125209763ÝÍã¤P™ ‚(:Å ¹æõÒ¬ýÅË) disk./.usagePercent‚76.14451155098034 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 7717.05078125 cpu‚25.614642478712504 proc.cpu.threads‚17 network.sent‚ 4.138929e+06% disk./.usageGB‚1169.0086135864258" disk.dm-1.out‚3658.72119140625 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme1c1n1.out‚ 7717.05078125 disk.md0.in‚0 disk.dm-0.out‚18578.08203125 network.recv‚646654 disk.dm-1.in‚0( disk.nvme0c0n1.out‚14683.39599609375& disk.nvme0n1.out‚14683.39599609375 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625' proc.memory.availableMB‚ 492.8203125 proc.memory.rssMB‚ 2987.09375 disk.nvme0n1.in‚ 0.00390625 disk.md0.out‚76.8125$ memory_percent‚96.99206352233887* proc.memory.percent‚18.231773376464844oGœ²žš ‚(:“ ¹æõÒ©Ý†Ì gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚273.775' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.25833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ïäµU› ‚(:Ê ÈæõÒüÄÎË( disk./.usagePercent‚76.1444807402029 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 disk.md0.out‚ 77.23828125$ memory_percent‚96.99063301086426 network.sent‚ 4.179944e+06 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚14684.37939453125$ disk.nvme1n1.out‚7717.6748046875 proc.cpu.threads‚17% disk./.usageGB‚1169.0081405639648 disk.dm-0.out‚ 18578.6953125" disk.dm-1.out‚3658.79541015625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚7717.6748046875 disk.nvme1n1.in‚0 disk.md0.in‚0' proc.memory.availableMB‚ 493.0546875 proc.memory.rssMB‚ 2987.09375* proc.memory.percent‚18.231773376464844 cpu‚25.61383580620773 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚14684.37939453125 network.recv‚650491 /ïøžœ ‚(:“ ÈæõÒ ‘Ì gpu.0.gpu‚87.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚267.408' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.13600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nI*2 - _wandbruntime‚2442  _runtime‚2442ì×àµÇž Á % _timestamp‚1.7845092689129689e+09 _runtime‚2442.488534047 _step‚222 & lr-AdamW/pg1‚0.0005929679724477357 & lr-AdamW/pg2‚0.0005929679724477357 trainer/global_step‚27490õlÌŸ Æ % _timestamp‚1.7845092689129689e+09 _runtime‚2442.488534047 _step‚222 & lr-AdamW/pg1‚0.0005929679724477357 & lr-AdamW/pg2‚0.0005929679724477357 trainer/global_step‚2749Þ_ø´&2  - _wandbruntime‚2442  _runtime‚2442™ìƒÒ¡ ÛO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚223 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3179288691200 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.frI…Ùÿ`"eed‚419444519197184 trainer/global_step‚2749 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27152196 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3179288691200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚422623824927744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3178947267072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27142053 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6568330 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3174479802506 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22171501 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚91509463279104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91509444666880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚416997417692560 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚422623824927744 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49323699 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22171501 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6568057 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3174139241232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22171503 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4798812 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚422623466464256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚419444536236544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3197729828864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3197626656768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚416997434731920 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419444519197184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27152196 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11366866 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94707193107968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420171556933792 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚422623466464256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22171503 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94707071323648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚419444536236544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4798809 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27142053 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚420171914534426 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49313554 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11367142 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3174139241232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22171503 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4798812 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚422623466464256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚419444536236544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3197729828864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3197626656768 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚416997434731920 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419444519197184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27152196 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11366866 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94707193107968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420171556933792 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚422623466464256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22171503 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94707071323648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚419444536236544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4798809 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27142053 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚420171914534426 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49313554 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11367142 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4800543 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚94741486163968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2443.093319379 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27161866 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚422773209536000 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419592810738688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚417144865399392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49341162 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27151709 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49341162 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11371127 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11370854 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚2749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3175588190870 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27161866 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 $ _timestamp‚1.784509269517545e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚422773209536000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22179294 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419592810738688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3180398797312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3198741547008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3175247629484 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚419592793699328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3180398797312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6570311 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22179296 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419592793699328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22179294 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6570581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49331003 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91542643549184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚91542624936960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4800546 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚422772851065344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11370854 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚2749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3175588190870 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27161866 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 $ _timestamp‚1.784509269517545e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚422773209536000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22179294 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419592810738688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3180398797312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3198741547008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3175247629484 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚419592793699328 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3180398797312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6570311 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 OADeviceStatsMonitor.on_train_batch_end/allocŠEùÛtated_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22179296 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419592793699328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22179294 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6570581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49331003 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91542643549184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚91542624936960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4800546 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚422772851065344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4800543 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚94741486163968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2443.093319379 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27161866 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚422773209536000 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419592810738688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚417144865399392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49341162 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0àgæ¡2¦ - _wandbruntime‚2443  _runtime‚2443*’µ§ € 0loss/FRt1/assignment_loss‚1.8687043402755565 2loss/SRqqt1/assignment_loss‚0.6017926974799673 1loss/FBt2/assignment_loss‚0.06458990185551323 2loss/FBt1/assignment_loss‚0.008513899119761262 %loss/total_loss‚8.940518494745637 epoch‚6 trainer/global_step‚2749 .loss/FRt2/detection_loss‚1.652963641826111 1loss/SRqqt1/detection_loss‚0.7378889377825231 1loss/SRqqt2/detection_loss‚0.7428565406906368 % _timestamp‚1.7845092695251443e+09 _runtime‚2443.100481878 0loss/FRt2/assignment_loss‚0.9564582743608129 2loss/SRqqt2/assignment_loss‚0.6187097525395574 /loss/FRt1/detection_loss‚0.5628314757523769 /loss/FBt1/detection_loss‚0.3782464250028637 .loss/FBt2/detection_loss‚0.746962608059957 _step‚225À; ¨ … 2loss/FBt1/assignment_loss‚0.008513899119761262 %loss/total_loss‚8.940518494745637 epoch‚6 trainer/global_step‚2749 .loss/FRt2/detection_loss‚1.652963641826111 1loss/SRqqt1/detection_loss‚0.7378889377825231 1loss/SRqqt2/detection_loss‚0.7428565406906368 % _timestamp‚1.7845092695251443e+09 _runtime‚2443.100481878 0loss/FRt2/assignment_loss‚0.9564582743608129 2loss/SRqqt2/assignment_loss‚0.6187097525395574 /loss/FRt1/detection_loss‚0.5628314757523769 /loss/FBt1/detection_loss‚0.3782464250028637 .loss/FBt2/detection_loss‚0.746962608059957 _step‚225 0loss/FRt1/assignment_loss‚1.8687043402755565 2loss/SRqqt1/assignment_loss‚0.6017926974799673 1loss/FBt2/assignment_loss‚0.06458990185551323áèŒÑXS© ‚(:È ×æõÒÖ›¢Ë proc.cpu.threads‚17" disk.dm-1.out‚3658.86572265625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚14686.306640625 network.sent‚ 4.183009e+06 network.recv‚653923) disk./.usagePercent‚76.12903683800162# disk./.usageGB‚1168.77103805542 disk.md0.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚7718.3857421875$ memory_percent‚96.99256420135498# disk.dm-0.out‚18580.25537109375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme0n1.out‚14686.306640625$ disk.nvme1n1.out‚7718.3857421875( proc.memory.availableMB‚ 492.73828125* proc.memory.percent‚18.231773376464844 cpu‚25.61304804019652 disk.md0.out‚ 77.703125 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 proc.memory.rssMB‚ 2987.09375D)§žª ‚(:“ ×æõÒ䚀Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚285.811' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.27033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05(ÂS« ‚(:È ææõÒ»áªË# disk.dm-0.out‚18581.37646484375 proc.memory.rssMB‚ 2987.09375* proc.memory.percent‚18.231773376464844 cpu‚25.61215986654821 proc.cpu.threads‚17 disk.nvme1n1.in‚0# disk.nvme1n1.out‚7719.208984375 disk.md0.in‚0 disk.md0.out‚ 78.03515625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme0c0n1.out‚14687.46484375 disk.nvme0n1.in‚ 0.00390625$ memory_percent‚96.99313640594482( proc.memory.availableMB‚ 492.64453125 disk.dm-1.in‚0" disk.dm-1.out‚3658.99560546875 network.recv‚658270) disk./.usagePercent‚76.12904205595588% disk./.usageGB‚1168.7711181640625 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0# disk.nvme0n1.out‚14687.46484375 network.sent‚ 4.247405e+06 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚7719.208984375PÆEž¬ ‚(:“ ææõÒú÷óË gpu.0.gpu‚93.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚146.167' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚48.72233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@G"S­ ‚(:È õæõÒ²ºË network.sent‚ 4.250327e+06( disk./.usagePercent‚76.1290828056938" disk.dm-1.out‚3662.28076171875% disk.nvme1n1.out‚7723.65478515625& proc.memory.availableMB‚ 492.640625 proc.memory.rssMB‚ 2987.09375 network.recv‚661747 disk.rbd0.in‚0' disk.nvme0c0n1.out‚14692.9833984375' disk.nvme1c1n1.out‚7723.65478515625* proc.memory.percent‚18.231773376464844 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚ 18586.5546875 disk.dm-1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 cpu‚25.611281187400575$ disk./.usageGB‚1168.771743774414 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚14692.9833984375 disk.nvme1c1n1.in‚0 disk.md0.out‚ 78.7421875$ memory_percent‚96.99316024780273¡o‘® ‚(:† õæõÒÛ‡Ì gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚121.188' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›.Çk2¯ - _wandbruntime‚2484  _runtime‚2484>¥ÊwÇ° Á & lr-AdamW/pg2‚0.0005920534591624983 trainer/global_step‚2799 % _timestamp‚1.7845093105741956e+09 _runtime‚2484.149843217 _step‚226 & lr-AdamW/pg1‚0.0005920534591624983 +Fq̱ Æ & lr-AdamW/pg1‚0.0005920534591624983 & lr-AdamW/pg2‚0.0005920534591624983 trainer/global_step‚2799 % _timestamp‚1.7845093105741956e+09 _runtime‚2484.149843217 _step‚226âd42² - _wandbruntime‚2484  _runtime‚2484CÄQß'³ ÙO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚430092760673280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22561151 _runtime‚ 2484.1511928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27636848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6682624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11567851 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27626705 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3253465712128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚426858261343744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3234857793024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚426858244304384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93168476783104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚424368951066160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27626705 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96421923883008 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚427598573520800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4885512 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22561153 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚427598931121434 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93168458170880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3253568884224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚430092760673280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚227 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚427598931121434 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93168458170880 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3253568884224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚430092760673280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚227 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚430092760673280 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22561151 _runtime‚ 2484.1511928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27636848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6682624 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11567851 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27626705 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145ãŸÇ^—2µ - _wandbruntime‚2484  _runtime‚24840͹ƒ¶ åM SEDeviceStatsMoniBÛ‚Ò&tor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22568944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4887246 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚228 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22568944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚427006518806528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93201657053184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3235626467840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4887243 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50215464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27646518 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96456345211392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3254572410368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427747470177270 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚424516381733632 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚430242503745024 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427006518806528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22568946 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3235626467840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427006535845888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50215464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3230730842892 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _runtime‚2484.754067313 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50205305 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3231071404278 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11571825 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96456210851328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚430242145274368 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27636361 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427006535845888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93201638440960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6684582 trainer/global_step‚2799 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22568946 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚427747112576524 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3235967899136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27646518 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50205305 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6684864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚424516398772992M\ ð&· êM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3231071404278 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11571825 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96456210851328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚430242145274368 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27636361 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427006535845888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93201638440960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6684582 trainer/global_step‚2799 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22568946 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚427747112576524 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3235967899136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27646518 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50205305 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6684864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚424516398772992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22568944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4887246 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚228 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22568944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚427006518806528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93201657053184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3235626467840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4887243 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50215464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27646518 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96456345211392 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3254572410368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427747470177270 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚424516381733632 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚430242503745024 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427006518806528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22568946 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3235626467840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚427006535845888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50215464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3230730842892 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _runtime‚2484.754067313 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50205305 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ä3é2¸ - _wandbruntime‚2484  _runtime‚2484¡<‰¹ þ ,loss/FRt2/detection_loss‚1.5829288333235 /loss/FBt1/detection_loss‚0.7107666990273587 /loss/FBt2/detection_loss‚0.4877667614137349 trainer/global_step‚2799 2loss/SRqqt1/assignment_loss‚0.6299737412272364 1loss/SRqqt1/detection_loss‚0.9478761302408113 epoch‚6 2loss/FBt1/assignment_loss‚0.010916576505224575 1loss/FBt2/assignment_loss‚0.07539227534994739 %loss/total_loss‚9.147365451558052 % _timestamp‚1.7845093111871805e+09 _runtime‚2484.762467906 _step‚229 0loss/FRt2/assignment_loss‚0.8945907743824502 2loss/SRqqt2/assignment_loss‚0.6556237423435889 /loss/FRt1/detection_loss‚0.4951142091490077 1loss/SRqqt2/detection_loss‚0.6963779538352889 /loss/FRt1/assignment_loss‚1.960037754759902³[£ º ƒ 0loss/FRt2/assignment_loss‚0.8945907743824502 2loss/SRqqt2/assignment_loss‚0.6556237423435889 /loss/FRt1/detection_loss‚0.4951142091490077 1loss/SRqqt2/detection_loss‚0.6963779538352889 /loss/FRt1/assignment_loss‚1.960037754759902 ,loss/FRt2/detection_loss‚1.5829288333235 /loss/FBt1/detection_loss‚0.7107666990273587 /loss/FBt2/detection_loss‚0.4877667614137349 trainer/global_step‚2799 2loss/SRqqt1/assignment_loss‚0.6299737412272364 1loss/SRqqt1/detection_loss‚0.9478761302408113 epoch‚6 2loss/FBt1/assignment_loss‚0.010916576505224575 1loss/FBt2/assignment_loss‚0.07539227534994739 %loss/total_loss‚9.147365451558052 % _timestamp‚1.7845093111871805e+09 _runtime‚2484.762467906 _step‚229å*ŽÅuV» ‚(:Ë „çõÒÛÜâË disk.nvme0n1.in‚ 0.00390625* proc.memory.percent‚18.239831924438477 disk.dm-1.in‚0 disk.dm-1.out‚3662.376953125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚14694.72802734375$ memory_percent‚97.00150489807129) disk./.usagePercent‚76.12909075686217 disk.md0.out‚ 79.9765625 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚ 18587.109375 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚14694.72802734375 cpu‚25.61599498087466' disk.nvme1c1n1.out‚7725.09716796875 disk.nvme1n1.in‚0% disk.nvme1n1.out‚7725.09716796875 disk.md0.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 491.2734375" proc.memory.rssMB‚ 2988.4140625 proc.cpu.threads‚17 network.sent‚ 4.269072e+06 network.recv‚664642% disk./.usageGB‚1168.7718658447266 disk.nvme1c1n1.in‚0,#”ž¼ ‚(:“ „çõÒ®‚§Ì gpu.0.gpu‚99.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚189.998' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.33266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Hø„2½ - _wandbruntime‚2500  _runtime‚25001²%š&¾ öz =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7181155681610107 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6098761558532715 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03611548990011215 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2753002345561981 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22366580367088318 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.08965524286031723 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3489583432674408 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8239647746086121 *jet/accuracy_1_of_3‚0.9051983952522278 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6082812547683716 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7354308366775513 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8"Purity/2FRt*SRqqt0FBt/event_purity‚0.024300467222929 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23567207157611847 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7280042171478271 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7478004097938538 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8606382012367249 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.726123571395874 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4333555996417999 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22289150953292847 particle/accuracy_3_of_3‚1 _runtime‚2500.288597584 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22476066648960114 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21180149912834167 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20806415379047394 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4658915102481842 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09191820025444031 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9665167927742004 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 (loss/val_total_loss‚8.73949876287414 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6188223958015442 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9253276586532593 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36464834213256836 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44618016481399536 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1998632550239563 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7717409729957581 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7359108924865723 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03435641527175903 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 epoch‚6 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9989115595817566 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2796213626861572 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10782698541879654 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966214299201965 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6051011085510254 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6098761558532715 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3073437213897705 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4318878948688507 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7242016792297363 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.048879582434892654 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7269387245178223 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6725841760635376 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03232342749834061 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7702183723449707 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7527361512184143 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3552101254463196 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.988857090473175 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17238695919513702 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_2_of_4‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18967017531394958 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16211268305778503 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6651447415351868 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18832415342330933 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7120978236198425 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.803911566734314 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22315336763858795 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5925782322883606 particle/accuracy_1_of_4‚1 /particle/accuracy_3_of_4‚0.9938492774963379 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6095868349075317 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6385558843612671 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6086215376853943 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03611548990011215 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.813378632068634 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9980952143669128 (particle/accuracy‚0.8475205302238464 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6289209723472595 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6455880403518677 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5950174927711487 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21180149912834167 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7375472187995911 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7175025343894958 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21732385456562042 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6558464765548706 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.65597ú„£Gâ89776802063 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7247215509414673 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_2‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21231785416603088 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.45285677909851074 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 +jet/accuracy_3_of_3‚0.06764090061187744 +particle/sensitivity‚0.6272457838058472 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20543216168880463 ,jet/accuracy_4_of_4‚0.011098379269242287 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8211073279380798 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7174131870269775 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7569644451141357 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6693216562271118 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36190176010131836 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8914162516593933 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27021703124046326 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6095921397209167 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03080029971897602 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4633159041404724 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10782124847173691 *jet/accuracy_2_of_4‚0.7616985440254211 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9281099438667297 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.62151038646698 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_5‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03844123333692551 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1835186928510666 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0601569227874279 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 jet/accuracy_2_of_5‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992558360099792 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2047223597764969 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03412381932139397 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 trainer/global_step‚2799 $ _timestamp‚1.784509326712619e+09 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20423632860183716 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8011337518692017 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12424477189779282 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5349999666213989 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9960024952888489 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5452924370765686 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8145880103111267 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 _step‚230 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6567996144294739 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6717791557312012 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5818617939949036 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5347502827644348 +jet/accuracy_2_of_2‚0.15326863527297974 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5345140099525452 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03203093260526657 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8239647746086121 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3828125 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9077574610710144 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9887582063674927 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_1‚0.3114096522331238 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17619502544403076 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930437803268433 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17704862356185913 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8673996925354004 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13833333551883698 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04757506400346756 *jet/accuracy_2_of_3‚0.5626760721206665 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912398457527161 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686¢ˆÚ=¿ ûz ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3828125 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9077574610710144 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9887582063674927 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_1‚0.3114096522331238 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17619502544403076 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930437803268433 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17704862356185913 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8673996925354004 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13833333551883698 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04757506400346756 *jet/accuracy_2_of_3‚0.5626760721206665 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912398457527161 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7181155681610107 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6098761558532715 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03611548990011215 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2753002345561981 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22366580367088318 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.08965524286031723 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3489583432674408 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8239647746086121 *jet/accuracy_1_of_3‚0.9051983952522278 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6082812547683716 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7354308366775513 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8"Purity/2FRt*SRqqt0FBt/event_purity‚0.024300467222929 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23567207157611847 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7280042171478271 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7478004097938538 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8606382012367249 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.726123571395874 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4333555996417999 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22289150953292847 particle/accuracy_3_of_3‚1 _runtime‚2500.288597584 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22476066648960114 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21180149912834167 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20806415379047394 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4658915102481842 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09191820025444031 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9665167927742004 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 (loss/val_total_loss‚8.73949876287414 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6188223958015442 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9253276586532593 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36464834213256836 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44618016481399536 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1998632550239563 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7717409729957581 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7359108924865723 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03435641527175903 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 epoch‚6 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9989115595817566 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2796213626861572 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10782698541879654 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966214299201965 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6051011085510254 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6098761558532715 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3073437213897705 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4318878948688507 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7242016792297363 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.048879582434892654 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7269387245178223 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6725841760635376 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03232342749834061 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7702183723449707 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7527361512184143 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3552101254463196 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.988857090473175 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17238695919513702 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_2_of_4‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18967017531394958 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16211268305778503 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6651447415351868 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18832415342330933 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7120978236198425 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.803911566734314 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22315336763858795 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5925782322883606 particle/accuracy_1_of_4‚1 /particle/accuracy_3_of_4‚0.9938492774963379 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6095868349075317 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6385558843612671 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6086215376853943 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03611548990011215 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.813378632068634 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9980952143669128 (particle/accuracy‚0.8475205302238464 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6289209723472595 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6455880403518677 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5950174927711487 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21180149912834167 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7375472187995911 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7175025343894958 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21732385456562042 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6558464765548706 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6559789776802063 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7247215509414673 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_2‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21231785416603088 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.45285677909851074 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 +jet/accuracy_3_of_3‚0.06764090061187744 +particle/sensitivity‚0.6272457838058472 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20543216168880463 ,jet/accuracy_4_of_4‚0.011098379269242287 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8211073279380798 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7174131870269775 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7569644451141357 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6693216562271118 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36190176010131836 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8914162516593933 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27021703124046326 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6095921397209167 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03080029971897602 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4633159041404724 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10782124847173691 *jet/accuracy_2_of_4‚0.7616985440254211 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9281099438667297 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.62151038646698 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_5‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03844123333692551 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1835186928510666 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0601569227874279 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 jet/accuracy_2_of_5‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992558360099792 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2047223597764969 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03412381932139397 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 trainer/global_step‚2799 $ _timestamp‚1.784509326712619e+09 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20423632860183716 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8011337518692017 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12424477189779282 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5349999666213989 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9960024952888489 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5452924370765686 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8145880103111267 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 _step‚230 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6567996144294739 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6717791557312012 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5818617939949036 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5347502827644348 +jet/accuracy_2_of_2‚0.15326863527297974 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5345140099525452 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03203093260526657 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8239647746086121 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625æoÙÜ\À ‚(:Ñ “çõÒç–ÍË network.sent‚ 4.330204e+06 network.recv‚668420 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚7973.77490234375 disk.md0.in‚0 disk.dm-1.out‚3907.822265625 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚14707.73095703125 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚18600.9736328125 disk.dm-1.in‚0$ memory_percent‚95.49915790557861( proc.memory.availableMB‚ 737.41796875# proc.memory.rssMB‚ 2997.74609375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚18.29679012298584) disk./.usagePercent‚76.12909249618026% disk./.usageGB‚1168.7718925476074% disk.nvme1n1.out‚7973.77490234375 disk.md0.out‚ 81.1171875 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚14707.73095703125 cpu‚25.625521797698735>E\mžÁ ‚(:“ “çõÒÎŽŽÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚268.794' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.59799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!vó\R ‚(:Ç ¢çõÒ«éµË network.sent‚ 4.375132e+06 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 82.32421875 disk.rbd0.in‚0$ disk.nvme0n1.out‚14866.080078125% proc.memory.availableMB‚ 737.65625 network.recv‚671566& disk.nvme1c1n1.out‚8122.8408203125" disk.dm-0.out‚18849.5009765625 disk.nvme0n1.in‚ 0.00390625 cpu‚25.624586161126917 disk.dm-1.out‚ 3964.16796875) disk./.usagePercent‚76.11359517206647% disk./.usageGB‚1168.5339698791504 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2997.74609375) proc.memory.percent‚18.29679012298584$ disk.nvme1n1.out‚8122.8408203125 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0c0n1.out‚14866.080078125# memory_percent‚95.4977035522461 proc.cpu.threads‚17œ’ˆ¸žà ‚(:“ ¢çõÒñ‘ÿË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚276.707' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.23566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gšéÄQÄ ‚(:Æ ±çõÒ“ö•Ë disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚8315.58837890625 disk.nvme1n1.in‚0$ memory_percent‚95.49775123596191' proc.memory.availableMB‚ 737.6484375# proc.memory.rssMB‚ 2997.74609375 network.sent‚ 4.378054e+06$ disk./.usageGB‚1168.534091949463% disk.nvme1n1.out‚8315.58837890625 disk.md0.out‚83.5625) proc.memory.percent‚18.29679012298584 proc.cpu.threads‚17 network.recv‚674858) disk./.usagePercent‚76.11360312323484 disk.dm-1.out‚ 4153.59375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚14882.27197265625 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚18866.42578125 disk.rbd0.in‚0& disk.nvme0n1.out‚14882.27197265625 disk.nvme1c1n1.in‚0 disk.md0.in‚0 cpu‚25.623564892099697 disk.dm-1.in‚06åœÅ ‚(:‘ ±çõÒÒæË gpu.0.gpu‚23.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.07' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.35666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌÝ@2Æ - _wandbruntime‚2542  _runtime‚2542F7˜ ÇÇ Á _runtime‚2542.419540947 _step‚231 & lr-AdamW/pg1‚0.0005911206531435709 & lr-AdamW/pg2‚0.0005911206531435709 trainer/global_step‚2849 % _timestamp‚1.7845093688439617e+09fä :ÌÈ Æ % _timestamp‚1.7845093688439617e+09 _runtime‚2542.419540947 _step‚231 & lr-AdamW/pg1‚0.0005911206531435709 & lr-AdamW/pg2‚0.0005911206531435709 trainer/global_step‚2849ç)qƒû2É - _wandbruntime‚2542  _runtime‚2542ð:•gtÊ ÚO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23073609 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436214330326528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23073611 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23073611 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚439524716596224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11812374 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94850068949504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3305750078224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3310403309056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚2849 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚439525075059712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28202999 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3310744733184 WKDeviceStatsMonitor.on_tr¯Vºl ain_batch_start/inactive_split.small_pool.allocated‚6836046 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98179979740160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51266465 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28192856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439524716596224 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3310403309056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51276610 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6835761 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11812086 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3330028380672 _runtime‚2542.420310949 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433661740447984 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23073609 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚436967149964934 % _timestamp‚1.7845093688448708e+09Z·œšå'Ë ßO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23073611 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚439524716596224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11812374 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚94850068949504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3305750078224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3310403309056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚2849 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚439525075059712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28202999 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3310744733184 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6836046 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98179979740160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51266465 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28192856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439524716596224 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3310403309056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51276610 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6835761 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11812086 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3330028380672 _runtime‚2542.420310949 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433661740447984 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23073609 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚436967149964934 % _timestamp‚1.7845093688448708e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23073609 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436214330326528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23073611èeãH62Ì - _wandbruntime‚2543  _runtime‚2543ó÷ݽê&Í äM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚436362604828672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3331044678144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 trainer/global_step‚2849 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3311513408000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51294073 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28212669 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433809188154816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433809171115456 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11816048 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94883230607360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3306517905202 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28212669 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6838259 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚437115689020658 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98214394965504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3311513408000 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _step‚233 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 trainer/global_step‚2849 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3311513408000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51294073 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28212669 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433809188154816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433809171115456 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11816048 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94883230607360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3306517905202 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28212669 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6838259 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚437115689020658 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98214394965504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3311513408000 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _step‚233 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99894812222976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28687651 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚443628055433728 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52150912 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23463259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52140767 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚443628038394368 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚96509082453504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚441033273821584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚446994369268736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚446994369268736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12013678 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12013396 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3385862039552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3385748381696 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3365972410880 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28687651 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚446994010805248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚441033290860944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23463261 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28677508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚99894944493056 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99894812222976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28687651 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚443628055433728 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52150912 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23463259 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52140767 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚443628038394368 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚96509082453504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚441033273821584 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚446994369268736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚446994369268736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12013678 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12013396 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3385862039552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3385748381696 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3365972410880 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28687651 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚446994010805248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚441033290860944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23463261 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28677508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚99894944493056 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚443776312896512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28697321 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28687164 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441180704489056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3362001118610 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚443776329935872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28687164 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99929233920000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447143395406336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.Шʈýcurrent‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23471054 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441180721528416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52158216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5064762 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚444542705607666 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52168375 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52168375 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚96542262723584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28697321 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447143753876992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12017419 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3386971196416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚444542705607666 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52168375 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52168375 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚96542262723584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28697321 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447143753876992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12017419 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3386971196416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚443776312896512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28697321 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28687164 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441180704489056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3362001118610 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚443776329935872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28687164 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99929233920000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447143395406336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23471054 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441180721528416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52158216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5064762 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452í}ÓôT2á - _wandbruntime‚2585  _runtime‚2585U^÷â € 2loss/SRqqt1/assignment_loss‚0.4853913538279396 2loss/SRqqt2/assignment_loss‚0.6073509668712019 2loss/FBt2/assignment_loss‚0.023317055398223374 /loss/FRt2/detection_loss‚1.6041839544562064 $ _timestamp‚1.784509411495194e+09 0loss/FRt2/assignment_loss‚0.8246806132309328 2loss/FBt1/assignment_loss‚0.029029639926806374 1loss/SRqqt1/detection_loss‚0.7714102599670976 1loss/SRqqt2/detection_loss‚0.7566256813617571 trainer/global_step‚2899 /loss/FRt1/detection_loss‚0.5457941395535056 epoch‚7 _runtime‚2585.071304549 0loss/FRt1/assignment_loss‚1.8762468689808407 .loss/FBt1/detection_loss‚0.459457652906044 /loss/FBt2/detection_loss‚1.1771769501053944 $loss/total_loss‚9.16066513658595 _step‚238Ýù©/ ã … 0loss/FRt1/assignment_loss‚1.8762468689808407 .loss/FBt1/detection_loss‚0.459457652906044 /loss/FBt2/detection_loss‚1.1771769501053944 $loss/total_loss‚9.16066513658595 _step‚238 2loss/SRqqt1/assignment_loss‚0.4853913538279396 2loss/SRqqt2/assignment_loss‚0.6073509668712019 2loss/FBt2/assignment_loss‚0.023317055398223374 /loss/FRt2/detection_loss‚1.6041839544562064 $ _timestamp‚1.784509411495194e+09 0loss/FRt2/assignment_loss‚0.8246806132309328 2loss/FBt1/assignment_loss‚0.029029639926806374 1loss/SRqqt1/detection_loss‚0.7714102599670976 1loss/SRqqt2/detection_loss‚0.7566256813617571 trainer/global_step‚2899 /loss/FRt1/detection_loss‚0.5457941395535056 epoch‚7 _runtime‚2585.071304549îž‘ØVä ‚(:Ë íçõÒ¤íÔË disk.md0.in‚0# disk.dm-0.out‚18882.76904296875 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 4.467863e+06% disk./.usageGB‚1168.5335273742676 disk.nvme1c1n1.in‚0 disk.md0.out‚ 87.0078125 disk.dm-1.out‚4154.080078125 disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚95.49846649169922% proc.memory.availableMB‚ 737.53125' disk.nvme1c1n1.out‚8321.59912109375 disk.dm-1.in‚0% disk.nvme0n1.out‚14900.3115234375 cpu‚25.61994963016444 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625' disk.nvme0c0n1.out‚14900.3115234375 disk.nvme0n1.in‚ 0.00390625# proc.memory.rssMB‚ 2997.74609375) proc.memory.percent‚18.29679012298584 network.recv‚689048( disk./.usagePercent‚76.1135663490811 disk.nvme1n1.in‚0% disk.nvme1n1.out‚8321.59912109375ü·,ä“å ‚(:ˆ íçõÒèçŽÌ gpu.0.gpu‚92.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚267.051' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.017 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!0„Ü]æ ‚(:Ò üçõÒóú¤Ë& disk.nvme0n1.out‚15034.04638671875 disk.nvme1n1.in‚0" disk.dm-0.out‚19048.1083984375$ memory_percent‚95.49849033355713 network.recv‚693201 disk.rbd0.out‚0( disk.nvme0c0n1.out‚15034.04638671875% disk.nvme1n1.out‚8354.22509765625( proc.memory.availableMB‚ 737.52734375# proc.memory.rssMB‚ 2997.74609375) proc.memory.percent‚18.29679012298584 proc.cpu.threads‚17) disk./.usagePercent‚76.14875002067303 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625 network.sent‚ 4.519112e+06% disk./.usageGB‚1169.0736846923828 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚8354.22509765625 disk.md0.in‚0 disk.md0.out‚ 87.45703125 cpu‚25.619176952023423 disk.dm-1.out‚4154.138671875Þ+ºúžç ‚(:“ üçõÒ‹ÍäË gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚265.816' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.60533333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cÌr9[è ‚(:Ð ‹èõÒëÜË$ memory_percent‚96.66707515716553 cpu‚25.61763939227415 proc.cpu.threads‚17 network.recv‚697034% disk./.usageGB‚1169.0098876953125 disk.md0.out‚ 87.82421875 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚19375.86865234375 disk.dm-1.in‚0 disk.dm-1.out‚4154.279296875 disk.nvme0c0n1.in‚ 0.00390625 network.sent‚ 4.523921e+06 disk.rbd0.out‚0' disk.nvme0c0n1.out‚15310.2236328125% disk.nvme0n1.out‚15310.2236328125 disk.nvme1n1.in‚0% disk.nvme1n1.out‚8406.73583984375( proc.memory.availableMB‚ 546.06640625) proc.memory.percent‚18.29679012298584 disk.md0.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚8406.73583984375# proc.memory.rssMB‚ 2997.74609375) disk./.usagePercent‚76.14459454130028[”—œé ‚(:‘ ‹èõÒßØ Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.986' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.32866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ&͉2ê - _wandbruntime‚2626  _runtime‚2626½í àÇë Á % _timestamp‚1.7845094531874826e+09 _runtime‚2626.765048766 _step‚239 & lr-AdamW/pg1‚0.0005892004037252565 & lr-AdamW/pg2‚0.0005892004037252565 trainer/global_step‚29496r¸›Ìì Æ _step‚239 & lr-AdamW/pg1‚0.0005892004037252565 & lr-AdamW/pg2‚0.0005892004037252565 trainer/global_step‚2949 % _timestamp‚1.7845094531874826e+09 _runtime‚2626.765048766ï秓2í - _wandbruntime‚2626  _runtime‚2626Š#‹â'î ÜO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3421882936832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3416375943766 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53015069 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451041780540928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚451041763501568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90589184 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454463305014272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3421541512704 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12214831 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29172303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451041763501568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29162160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23852911 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23852909 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12215120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53025214 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚451821540739584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚448404824234544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚101609688319488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3416716505040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23852911 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7065392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29172303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚448404807195184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3421541512704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3441610974208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3421882936832 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29162160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚451821183138950 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5149725 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451041780540928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98168095957504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23852909 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454463663477760 _runtime‚2626.765117036 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454463305014272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5149728 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7065106 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109201408 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98168077345280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚454463305014272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3421541512704 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12214831 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29172303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451041763501568 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29162160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23852911 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23852909 QDDeviceStatsMonitor.on_train_batch_start/inactivû&B e_split.all.allocated‚12215120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53025214 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚451821540739584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚448404824234544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚101609688319488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3416716505040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23852911 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7065392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29172303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚448404807195184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3421541512704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3441610974208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3421882936832 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29162160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚451821183138950 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5149725 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451041780540928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98168095957504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23852909 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454463663477760 _runtime‚2626.765117036 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454463305014272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5149728 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7065106 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109201408 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98168077345280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29181973 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚451190038003712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451190038003712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3417484332018 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454612689615360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53032518 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454613048086016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚451970079795420 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451190055043072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3422993042944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7067403 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53042677 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12218862 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845094537975662e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3442826039296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3422993042944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5151459 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451190055043072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98201276227584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98201257615360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2627.373303797 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚2949óô¡mò&ò ìM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451190038003712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3417484332018 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454612689615360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53032518 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454613048086016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚451970079795420 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚451190055043072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3422993042944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7067403 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53042677 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12218862 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845094537975662e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3442826039296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3422993042944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5151459 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚451190055043072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98201276227584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98201257615360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2627.373303797 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚2949 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29171816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12219131 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29181973 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚451190038003712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528ñ_³­-2ó - _wandbruntime‚2627  _runtime‚2627"Ì%ò ô † 2loss/SRqqt1/assignment_loss‚0.5117480919466606 /loss/FRt2/detection_loss‚1.6890143583904338 1loss/SRqqt2/detection_loss‚0.6375878846429917 % _timestamp‚1.7845094538005764e+09 1loss/SRqqt1/detection_loss‚0.7983744441867308 epoch‚7 _runtime‚2627.375940542 _step‚242 /loss/FBt2/detection_loss‚0.7078024476105648 %loss/total_loss‚8.631718074764512 0loss/FRt2/assignment_loss‚0.7877837449383337 2loss/SRqqt2/assignment_loss‚0.5906899229408458 2loss/FBt1/assignment_loss‚0.008490497644765557 4loss/FBt2/assignment_loss‚0.00031989276267025036 /loss/FBt1/detection_loss‚0.5512959082261659 trainer/global_step‚2949 0loss/FRt1/assignment_loss‚1.8503535041919332 0loss/FRt1/detection_loss‚0.498257377282416875`õ ‹ /loss/FRt2/detection_loss‚1.6890143583904338 1loss/SRqqt2/detection_loss‚0.6375878846429917 % _timestamp‚1.7845094538005764e+09 1loss/SRqqt1/detection_loss‚0.7983744441867308 epoch‚7 _runtime‚2627.375940542 _step‚242 /loss/FBt2/detection_loss‚0.7078024476105648 %loss/total_loss‚8.631718074764512 0loss/FRt2/assignment_loss‚0.7877837449383337 2loss/SRqqt2/assignment_loss‚0.5906899229408458 2loss/FBt1/assignment_loss‚0.008490497644765557 4loss/FBt2/assignment_loss‚0.00031989276267025036 /loss/FBt1/detection_loss‚0.5512959082261659 trainer/global_step‚2949 0loss/FRt1/assignment_loss‚1.8503535041919332 0loss/FRt1/detection_loss‚0.49825737728241687 2loss/SRqqt1/assignment_loss‚0.5117480919466606òôc˜ Uö ‚(:Ê šèõÒã¿âË' disk.nvme1c1n1.out‚8407.30419921875 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1n1.out‚8407.30419921875 network.sent‚ 4.539733e+06( disk./.usagePercent‚76.1445950382483$ disk./.usageGB‚1169.009895324707 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚19376.40771484375& disk.nvme0c0n1.out‚15310.939453125 disk.dm-1.in‚0! disk.dm-1.out‚4154.3818359375# proc.memory.rssMB‚ 2997.74609375) proc.memory.percent‚18.29679012298584 cpu‚25.61650356704172 proc.cpu.threads‚17 disk.md0.out‚ 88.1171875 network.recv‚700470 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme0n1.out‚15310.939453125$ memory_percent‚97.00155258178711& proc.memory.availableMB‚ 491.265625 disk.nvme1c1n1.in‚0˜Š‘ “÷ ‚(:ˆ šèõÒƒƒ¥Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.983' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.661 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š55 Yø ‚(:Î ©èõÒ©ÙØË) proc.memory.percent‚18.29679012298584 proc.cpu.threads‚17% disk.nvme0n1.out‚15322.8388671875 network.recv‚704714% disk./.usageGB‚1169.0102500915527! disk.dm-1.out‚4154.4208984375 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.md0.out‚ 88.546875 cpu‚25.615757332326464 disk.rbd0.out‚0 disk.md0.in‚0# disk.dm-0.out‚19389.95263671875 disk.nvme0n1.in‚ 0.00390625 network.sent‚ 4.591066e+06( disk./.usagePercent‚76.1446181463314 disk.rbd0.in‚0' disk.nvme0c0n1.out‚15322.8388671875% disk.nvme1n1.out‚8409.91650390625$ memory_percent‚97.00160026550293# proc.memory.rssMB‚ 2997.74609375 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚8409.91650390625 disk.dm-0.in‚ 0.00390625' proc.memory.availableMB‚ 491.2578125°Èžù ‚(:“ ©èõÒÌ×—Ì gpu.0.gpu‚94.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚290.716' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.90533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]]§™2ú - _wandbruntime‚2668  _runtime‚2668Ùý—¦Çû Á & lr-AdamW/pg2‚0.0005882130836599014 trainer/global_step‚2999 % _timestamp‚1.7845094953275795e+09 _runtime‚2668.903167995 _step‚243 & lr-AdamW/pg1‚0.0005882130836599014ôbO$ü Æ _runtime‚2668.903167995 a¾ýä¨ _step‚243 & lr-AdamW/pg1‚0.0005882130836599014 & lr-AdamW/pg2‚0.0005882130836599014 trainer/global_step‚2999 % _timestamp‚1.7845094953275795e+09ó¦œÍƒ2ý - _wandbruntime‚2668  _runtime‚2668IvùJâ'þ ÜO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3477110614528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99827109461504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29646812 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚458455488608768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12416231 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459248199725958 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29656955 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚461932599223296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29656955 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103324541212672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24242559 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3471859157174 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29646812 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7179803 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚2668.906998811 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3497450363392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚455776357608144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3472199718448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 $ _timestamp‚1.784509495328501e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24242561 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚461932599223296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7179516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53889371 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3477452038656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5236428 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53899516 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2999 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459248557326592 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24242559 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458455505648128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455776340568784 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458455505648128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3477110614528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚461932957686784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24242561 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3477452038656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12415941 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896’=D ç'ÿ áO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3477110614528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚461932957686784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24242561 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3477452038656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12415941 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3477110614528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99827109461504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29646812 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚458455488608768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12416231 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459248199725958 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29656955 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚461932599223296 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29656955 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103324541212672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24242559 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3471859157174 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29646812 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7179803 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚2668.906998811 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3497450363392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚455776357608144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3472199718448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 $ _timestamp‚1.784509495328501e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24242561 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚461932599223296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7179516 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53889371 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3477452038656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5236428 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53899516 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2999 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459248557326592 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24242559 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458455505648128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455776340568784 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458455505648128ôÙ–¯2€ - _wandbruntime‚2669  _runtime‚2669-2Òí& çM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12419913 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24250354 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459396738781682 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2669.513334932 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7182052 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚462082342295040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24250354 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458603763110912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53916979 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5238159 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455923788275616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455923771236256 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29666625 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚462081983824384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 % _timestamp‚1.7845094959376132e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29666625 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458603780150272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53906820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29656468 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3498677191680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3478220713472 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3498563540992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3473308106812 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3472967545426 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3478562144768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3478220713472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53906820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458603763110912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24250352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99860289731584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103358966923264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24250352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103358834660352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53916979 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53906820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458603763110912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24250352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99860289731584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚103358966923264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24250352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103358834660352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53916979 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12419913 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24250354 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459396738781682 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2669.513334932 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7182052 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚462082342295040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24250354 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚458603763110912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53916979 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5238159 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455923788275616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455923771236256 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29666625 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚462081983824384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 % _timestamp‚1.7845094959376132e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29666625 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458603780150272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53906820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29656468 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3498677191680 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3478220713472 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3498563540992 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3473308106812 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3472967545426 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3478562144768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3478220713472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚465869213715968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24632209 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3553267754496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 119949312 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30131464 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30131464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105039372107776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30141607 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469401893432320 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24632211 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7294426 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚466675216312966 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105039510669312 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3533021140480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101486104353280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784509537703103e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24632211 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚469402251895808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54773818 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30141607 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784509537703103e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24632211 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚469402251895808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54773818 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30141607 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚465869213715968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24632209 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3553267754496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 119949312 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 3ªý„ò*57600634 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30131464 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30131464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105039372107776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30141607 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469401893432320 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24632211 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7294426 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚466675216312966 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105039510669312 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3533021140480ø+R¿Q2’ - _wandbruntime‚2711  _runtime‚2711)y-”î&“ èM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24640004 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚466017488218112 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚466823755368690 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3533789815296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3533789815296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7296398 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3534131246592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚466017488218112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30151277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3554391450624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54791281 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7296671 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12621257 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚469551636504064 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚466017505257472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3554496712704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚249 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54781122 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463295304609856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24640004 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5324859 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3534131246592 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5324862 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚466824112969436 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54791281 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105073799948288 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚3049 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101519303235584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30141120 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54781122 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101519284623360 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5324859 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3534131246592 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5324862 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚466824112969436 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54791281 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105073799948288 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚3049 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101519303235584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30141120 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54781122 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101519284623360 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3554391450624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54791281 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7296671 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12621257 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚469551636504064 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚466017505257472 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3554496712704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚249 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54781122 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463295304609856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24640004ùГÕK2• - _wandbruntime‚2711  _runtime‚2711ïЦ– þ 2loss/FBt1/assignment_loss‚0.004132382231265618 3loss/FBt2/assignment_loss‚0.0032203107466722775 /loss/FRt2/detection_loss‚1.6044659927971496 1loss/SRqqt2/detection_loss‚0.6688990333962633 %loss/total_loss‚8.374989904592487 epoch‚7 1loss/SRqqt1/detection_loss‚0.7684220210731735 /loss/FBt1/detection_loss‚0.5443322401799917 trainer/global_step‚3049 /loss/FRt1/detection_loss‚0.5011560239563136 /loss/FBt2/detection_loss‚0.3684343682094054 " _timestamp‚1.7845095383149e+09 _runtime‚ 2711.89034594 _step‚250 0loss/FRt1/assignment_loss‚1.9013716177968725 0loss/FRt2/assignment_loss‚0.8352757803931635 2loss/SRqqt1/assignment_loss‚0.5483266841725256 0loss/SRqqt2/assignment_loss‚0.62695344963969à ̎ — ƒ /loss/FRt1/detection_loss‚0.5011560239563136 /loss/FBt2/detection_loss‚0.3684343682094054 " _timestamp‚1.7845095383149e+09 _runtime‚ 2711.89034594 _step‚250 0loss/FRt1/assignment_loss‚1.9013716177968725 0loss/FRt2/assignment_loss‚0.8352757803931635 2loss/SRqqt1/assignment_loss‚0.5483266841725256 0loss/SRqqt2/assignment_loss‚0.62695344963969 2loss/FBt1/assignment_loss‚0.004132382231265618 3loss/FBt2/assignment_loss‚0.0032203107466722775 /loss/FRt2/detection_loss‚1.6044659927971496 1loss/SRqqt2/detection_loss‚0.6688990333962633 %loss/total_loss‚8.374989904592487 epoch‚7 1loss/SRqqt1/detection_loss‚0.7684220210731735 /loss/FBt1/detection_loss‚0.5443322401799917 trainer/global_step‚3049úí]ÅW˜ ‚(:Ì åèõÒ¬âèË disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625( proc.memory.availableMB‚ 491.15234375# proc.memory.rssMB‚ 2997.74609375 cpu‚25.61257112411383 disk.nvme1n1.in‚0% disk.nvme1n1.out‚8413.86669921875 disk.dm-1.out‚ 4154.82421875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚15332.77099609375 network.sent‚ 4.679044e+06( disk./.usagePercent‚76.1137139426441 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625& disk.nvme0n1.out‚15332.77099609375 proc.cpu.threads‚17 network.recv‚719269' disk.nvme1c1n1.out‚8413.86669921875 disk.md0.out‚ 90.4296875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625# memory_percent‚97.0022439956665) proc.memory.percent‚18.29679012298584% disk./.usageGB‚1168.5357933044434 disk.md0.in‚0! disk.dm-0.out‚19399.447265625b=ª™ ‚(:’ åèõÒþ¨Ì gpu.0.gpu‚24.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚289.836' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.61200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñà91Wš ‚(:Ì ôèõÒ„®Ë disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 8414.62109375" disk.nvme1n1.out‚ 8414.62109375 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.rbd0.out‚0 network.sent‚ 4.728486e+06 network.recv‚723264) disk./.usagePercent‚76.09823202391902 disk.nvme0n1.in‚ 0.00390625" disk.dm-1.out‚4154.93701171875$ memory_percent‚97.00238704681396# proc.memory.rssMB‚ 2997.74609375) proc.memory.percent‚18.29679012298584( disk.nvme0c0n1.out‚15333.86962890625& disk.nvme0n1.out‚15333.86962890625 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.md0.in‚0 disk.md0.out‚ 90.9140625! disk.dm-0.out‚19400.169921875( proc.memory.availableMB‚ 491.12890625 cpu‚25.611783142473076 proc.cpu.threads‚17% disk./.usageGB‚1168.2981071472168VŒ…ñ› ‚(:’ ôèõÒã¸þË gpu.0.gpu‚6.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.018' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.339333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡’ËMœ ‚(: ƒéõÒëó´Ë$ memory_percent‚97.00250625610352) proc.memory.percent‚18.29679012298584 cpu‚25.61108430412779 proc.cpu.threads‚17 network.recv‚726655) disk./.usagePercent‚76.09823326628909% disk./.usageGB‚1168.2981262207031 disk.md0.in‚0 disk.md0.out‚ 91.30859375 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 4.73517e+06% disk.nvme1n1.out‚8416.13525390625" disk.dm-0.out‚19410.8330078125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme0n1.out‚ 15343.9375& proc.memory.availableMB‚ 491.109375# proc.memory.rssMB‚ 2997.74609375' disk.nvme1c1n1.out‚8416.13525390625 disk.nvme1n1.in‚0 disk.nvme1c1n1.in‚0" disk.dm-1.out‚4155.01513671875 disk.nvme0c0n1.in‚ 0.00390625! disk.nvme0c0n1.out‚ 15343.9375Ç8¯ž ‚(:“ ƒéõÒì»òË gpu.0.gpu‚80.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚293.422' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.80733333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥_þò2ž - _wandbruntime‚2753  _runtime‚2753¸fîþÅŸ ¿ & lr-AdamW/pg1‚0.0005861843698813364 & lr-AdamW/pg2‚0.0005861843698813364 trainer/global_step‚3099 $ _timestamp‚1.784509579841942e+09 _runtime‚ 2753.41750226 _step‚251#mHEÊ  Ä _runtime‚ 2753.41750226 _step‚251 & lr-AdamW/pg1‚0.0005861843698813364 & lr-AdamW/pg2‚0.0005861843698813364 trainer/global_step‚3099 $ _timestamp‚1.784509579841942e+09û†‘Â;2¡ - _wandbruntime‚2753  _runtime‚2753„s›h@¢ ÜO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103145136469504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3609161605120 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30626259 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30616116 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3588590242304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3583166145264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5409828 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3588248818176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25021861 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚476871187641344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚473282955862528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3609050044416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚470519424355344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30626259 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55637975 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3588590242304 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2753.418240341 _step‚252 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚3099mb1ç'£ áO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30626259 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30616116 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3588590242304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3583166145264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5409828 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3588248818176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25021861 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚476871187641344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚473282955862528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3609050044416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚470519424355344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30626259 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55637975 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3588590242304 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2753.418240341 _step‚252 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚3099 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚103145136469504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3609161605120 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ü*;I›2¤ - _wandbruntime‚2754  _runtime‚2754–Jåï&¥ éM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106788458277888 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30635929 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚474250771955698 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚3099 _runtime‚2754.031826666 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30625772 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3610277995520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3610160150528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55665583 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477020572242432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25029654 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103178316739584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477020930713088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3589358917120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106788594735104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12822253 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3583933972242 % _timestamp‚1.7845095804560885e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5411559 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473431230364672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477020930713088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚470666855022816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3589700348416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55655424 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30635929 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚474250771955698 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚3099 _runtime‚2754.031826666 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30625772 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3610277995520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3610160150528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55665583 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477020572242432 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25029654 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103178316739584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477020930713088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3589358917120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106788594735104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12822253 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3583933972242 % _timestamp‚1.7845095804560885e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5411559 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473431230364672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477020930713088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚470666855022816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3589700348416ý^LÐÑ2§ - _wandbruntime‚2754  _runtime‚2754)8R¾¨ € 1loss/SRqqt1/detection_loss‚0.6820039555069198 /loss/FBt2/detection_loss‚0.6767570920288563 epoch‚7 % _timestamp‚1.7845095804570324e+09 0loss/FRt1/assignment_loss‚1.8367011455822533 2loss/SRqqt2/assignment_loss‚0.5712170780980828 _runtime‚2754.032384597 _step‚254 -loss/FBt1/detection_loss‚0.53839653543844 %loss/total_loss‚8.855047570776446 2loss/SRqqt1/assignment_loss‚0.6177502078596866 2loss/FBt1/assignment_loss‚0.056247669060575975 1loss/FBt2/assignment_loss‚0.03258788436164145 1loss/SRqqt2/detection_loss‚0.7494553083210362 trainer/global_step‚3099 0loss/FRt2/assignment_loss‚0.9243964932181619 /loss/FRt1/detection_loss‚0.5066131826768947 /loss/FRt2/detection_loss‚1.6629210186238956pÉ9g © … epoch‚7 % _timestamp‚1.7845095804570324e+09 0loss/FRt1/assignment_loss‚1.8367011455822533 2loss/SRqqt2/assignment_loss‚0.5712170780980828 _runtime‚2754.032384597 _step‚254 -loss/FBt1/detection_loss‚0.53839653543844 %loss/total_loss‚8.855047570776446 2loss/SRqqt1/assignment_loss‚0.6177502078596866 2loss/FBt1/assignment_loss‚0.056247669060575975 1loss/FBt2/assignment_loss‚0.03258788436164145 1loss/SRqqt2/detection_loss‚0.7494553083210362 trainer/global_step‚3099 0loss/FRt2/assignment_loss‚0.9243964932181619 /loss/FRt1/detection_loss‚0.5066131826768947 /loss/FRt2/detection_loss‚1.6629210186238956 1loss/SRqqt1/detection_loss‚0.6820039555069198 /loss/FBt2/detection_loss‚0.6767570920288563þ n™Uª ‚(:Ê ’éõÒãôúË disk.dm-1.out‚ 4155.171875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.md0.out‚91.625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚15344.74072265625$ memory_percent‚97.00448513031006( proc.memory.availableMB‚ 490.78515625# proc.memory.rssMB‚ 2997.74609375% disk./.usageGB‚1168.2975425720215 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 cpu‚25.610293464997334 network.recv‚729935) disk./.usagePercent‚76.09819524976528 disk.nvme0n1.in‚ 0.00390625 disk.md0.in‚0) proc.memory.percent‚18.29679012298584 proc.cpu.threads‚17 network.sent‚ 4.750941e+06& disk.nvme0n1.out‚15344.74072265625' disk.nvme1c1n1.out‚8416.73193359375% disk.nvme1n1.out‚8416.73193359375" disk.dm-0.out‚19411.3837890625­*/¶ž« ‚(:“ ’éõÒŽ½ºÌ gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚291.367' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.12233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑÌŽÌM¬ ‚(: ¡éõÒÙµÓË& disk.nvme0n1.out‚15359.50927734375% disk.nvme1n1.out‚8419.53564453125 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚8419.53564453125 disk.rbd0.out‚0 network.sent‚ 4.80049e+06 disk.md0.in‚0" disk.dm-0.out‚19428.0947265625$ memory_percent‚97.00298309326172% proc.memory.availableMB‚ 491.03125# proc.memory.rssMB‚ 2997.74609375) proc.memory.percent‚18.29679012298584) disk./.usagePercent‚76.10927594831995$ disk./.usageGB‚1168.467658996582 disk.nvme1n1.in‚0 disk.dm-1.out‚ 4155.22265625 disk.nvme0n1.in‚ 0.00390625 cpu‚25.60952757011805 proc.cpu.threads‚17 network.recv‚734198 disk.md0.out‚92 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚15359.50927734375Y@‚­ ‚(:’ ¡éõÒ•ùÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚274.92' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.64000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 F6^® ‚(:Ó °éõÒ€“ÎË network.recv‚737620 disk.md0.out‚ 92.42578125" disk.dm-0.out‚20158.2880859375( disk.nvme0c0n1.out‚15878.87744140625& disk.nvme0n1.out‚15878.87744140625) proc.memory.percent‚18.29679012298584 network.sent‚ 4.807332e+06 disk.md0.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚4155.4072265625 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1n1.out‚8631.46533203125$ memory_percent‚97.00601100921631) disk./.usagePercent‚76.14464622389472% disk./.usageGB‚1169.0106811523438 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚8631.46533203125# proc.memory.rssMB‚ 2997.74609375 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 490.53515625 cpu‚25.608942852571307 proc.cpu.threads‚17Æéಜ¯ ‚(:‘ °éõÒ°úÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚278.2' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.73333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÛÚ2° - _wandbruntime‚2795  _runtime‚2795&=ìDZ Á % _timestamp‚1.7845096217048903e+09 _runtime‚2795.280526212 _step‚255 & lr-AdamW/pg1‚0.0005851431064686218 & lr-AdamW/pg2‚0.0005851431064686218 trainer/global_step‚3149”Zÿ̲ Æ trainer/global_step‚3149 % _timestamp‚1.7845096217048903e+09 _runtime‚2795.280526212 _step‚255 & lr-AdamW/pg1‚0.0005851431064686218 & lr-AdamW/pg2‚0.0005851431064686218ÿ½oB2³ - _wandbruntime‚2795  _runtime‚2795ê$^ä'´ ÞO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56512277 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3643817920000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2795.281429704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108469151496192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7523690 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚484340481850368 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5496525 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104804149973504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31100768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚477890957728944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚484340481850368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3644159344128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚484340840313856 % _timestamp‚1.7845096217059531e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚480696680969728 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚480696663930368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7523415 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104804131361280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚484340840313856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚481529607087616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚3149 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚481529249486982 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25411509 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107104256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚480696680969728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3638308797398 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5496528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3643817920000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25411511 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚477890940689584 _step‚256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85—Þüé'µ ãO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5496528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3643817920000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25411511 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚477890940689584 _step‚256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56512277 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3643817920000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2795.281429704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108469151496192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7523690 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚484340481850368 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5496525 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104804149973504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31100768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚477890957728944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚581 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚484340481850368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3644159344128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚484340840313856 % _timestamp‚1.7845096217059531e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚480696680969728 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚480696663930368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7523415 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104804131361280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚484340840313856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚481529607087616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚3149 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚481529249486982 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25411509 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107104256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚480696680969728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3638308797398 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647€!öéO2¶ - _wandbruntime‚2795  _runtime‚2795~߈?±· çM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7525976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7845096223151364e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25419304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13024238 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5498262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104837330243584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25419302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3666126168576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56529726 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5498259 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56539885 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3644928018944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚480844938432512 trainer/global_step‚3149 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56529726 UCDeviceStatsMonitor.on_train_batch_end/actiÛH&<ve_bytes.small_pool.freed‚ 3644928018944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚484489866451456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7525696 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108503332537856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚484490224922112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3645269450240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3666020906496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚478038388396416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13023955 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56539885 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31110424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31120581 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3645269450240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25419302 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31110424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108503456412160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚481678146143452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚480844938432512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452QÅöQò&¸ ìM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56529726 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3644928018944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚484489866451456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7525696 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108503332537856 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚484490224922112 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3645269450240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3666020906496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚478038388396416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13023955 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56539885 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31110424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚581 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31120581 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3645269450240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25419302 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31110424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108503456412160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚481678146143452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚480844938432512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7525976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7845096223151364e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25419304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13024238 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5498262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104837330243584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25419302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3666126168576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56529726 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5498259 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56539885 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3644928018944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚480844938432512 trainer/global_step‚3149ÄZdô2¹ - _wandbruntime‚2795  _runtime‚2795vÕH„º  0loss/FRt1/assignment_loss‚1.8433660422964069 0loss/FRt2/assignment_loss‚0.9373385826009055 2loss/SRqqt1/assignment_loss‚0.5682390959758616 2loss/FBt1/assignment_loss‚0.014565941097946244 /loss/FRt2/detection_loss‚1.6114371062349313 trainer/global_step‚3149 % _timestamp‚1.7845096223176067e+09 2loss/SRqqt2/assignment_loss‚0.5405580955302923 2loss/FBt2/assignment_loss‚0.007185156042842023 1loss/SRqqt1/detection_loss‚0.7624815755144505 1loss/SRqqt2/detection_loss‚0.7000205599526189 $loss/total_loss‚8.54299494172669 /loss/FBt2/detection_loss‚0.5197901157092075 epoch‚7 _runtime‚2795.892912383 _step‚258 .loss/FRt1/detection_loss‚0.534889057354407 /loss/FBt1/detection_loss‚0.5031236134168199ÛËù8 » † $loss/total_loss‚8.54299494172669 /loss/FBt2/detection_loss‚0.5197901157092075 epoch‚7 _runtime‚2795.892912383 _step‚258 .loss/FRt1/detection_loss‚0.534889057354407 /loss/FBt1/detection_loss‚0.5031236134168199 0loss/FRt1/assignment_loss‚1.8433660422964069 0loss/FRt2/assignment_loss‚0.9373385826009055 2loss/SRqqt1/assignment_loss‚0.5682390959758616 2loss/FBt1/assignment_loss‚0.014565941097946244 /loss/FRt2/detection_loss‚1.6114371062349313 trainer/global_step‚3149 % _timestamp‚1.7845096223176067e+09 2loss/SRqqt2/assignment_loss‚0.5405580955302923 2loss/FBt2/assignment_loss‚0.007185156042842023 1loss/SRqqt1/detection_loss‚0.7624815755144505 1loss/SRqqt2/detection_loss‚0.7000205599526189‚VÃuÀV¼ ‚(:Ë ¿éõÒÉúµÌ' disk.nvme1c1n1.out‚8635.62255859375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2997.74609375 cpu‚25.608436811647643( disk.nvme0c0n1.out‚15894.51708984375 disk.nvme1n1.in‚0" disk.dm-0.out‚20177.0166015625! disk.dm-1.out‚4155.5048828125 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.md0.out‚ 92.8828125 network.recv‚740186 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0$ proc.memory.availableMB‚490.9375) proc.memory.percent‚18.29679012298584) disk./.usagePercent‚76.14465019947892% disk.nvme1n1.out‚8635.62255859375 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0$ memory_percent‚97.00355529785156 network.sent‚ 4.822186e+06" disk./.usageGB‚1169.0107421875 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚15894.51708984375Ì&☓½ ‚(:ˆ ¿éõÒ¹Ÿ¯Î gpu.0.gpu‚81.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚175.629' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚58.543 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žGY½Q¾ ‚(:Æ ÎéõÒ´ÓÛË proc.memory.availableMB‚491# proc.memory.rssMB‚ 2997.74609375 disk.nvme1c1n1.in‚0" disk.dm-0.out‚20177.6025390625 cpu‚25.607788510429845 disk.dm-0.in‚ 0.00390625 network.sent‚ 4.873581e+06 network.recv‚744373% disk./.usageGB‚1169.0107040405273& disk.nvme0n1.out‚15895.41552734375$ disk.nvme1n1.out‚8636.2978515625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.md0.out‚ 93.27734375" disk.dm-1.out‚4155.64111328125( disk.nvme0c0n1.out‚15895.41552734375" memory_percent‚97.003173828125) proc.memory.percent‚18.29679012298584 proc.cpu.threads‚17( disk./.usagePercent‚76.1446477147388& disk.nvme1c1n1.out‚8636.2978515625 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0Np,:œ¿ ‚(:‘ ÎéõÒµ™Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.297' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.43233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZDàLÀ ‚(:Á ÝéõÒ©´ÁË disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.md0.out‚ 93.8671875" memory_percent‚97.003173828125) proc.memory.percent‚18.29679012298584 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625% disk.nvme1c1n1.out‚8637.095703125 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2997.74609375 network.recv‚747837) disk./.usagePercent‚76.14465293269305 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚15897.0986328125% disk.nvme0n1.out‚15897.0986328125 disk.nvme1c1n1.in‚0 proc.memory.availableMB‚491 network.sent‚ 4.878377e+06" disk.dm-1.out‚4155.68017578125 disk.rbd0.in‚0# disk.nvme1n1.out‚8637.095703125 disk.md0.in‚0 cpu‚25.607313007133907# disk./.usageGB‚1169.01078414917# disk.dm-0.out‚20178.78466796875e‡D!žÁ ‚(:“ ÝéõÒ­€ƒÌ gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.653' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.21766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñª2 - _wandbruntime‚2837  _runtime‚2837n¡àÇà Á _runtime‚2837.788207427 _step‚259 & lr-AdamW/pg1‚0.0005840839941612578 & lr-AdamW/pg2‚0.0005840839941612578 trainer/global_step‚3199 % _timestamp‚1.7845096642126205e+09…iÌÄ Æ & lr-AdamW/pg1‚0.0005840839941612578 & lr-AdamW/pg2‚0.0005840839941612578 trainer/global_step‚3199 % _timestamp‚1.7845096642126205e+09 _runtime‚2837.788207427 _step‚259ƒø0r°2Å - _wandbruntime‚2837  _runtime‚2837y;2uã'Æ ÝO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488110406076928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚485262474063184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491810134522880 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110183873468928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚488956266073990 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3699387021824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25801161 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106463144865280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 trainer/global_step‚3199 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491809776059392 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚488956623674624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 # _timestamp‚1.78450966421353e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31585420 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13221012 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚488110389037568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491809776059392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57386579 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31595563 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106463163477504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚485262491102544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25801161 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚488110406076928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110184001544704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3699728445952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25801159 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3720838067200 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584•”ðZ|Ç âO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57396724 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_r€“öl&etries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110183873468928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚488956266073990 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3699387021824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25801161 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106463144865280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 trainer/global_step‚3199 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚491809776059392 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚488956623674624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 # _timestamp‚1.78450966421353e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31585420 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13221012 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚488110389037568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491809776059392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57386579 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31595563 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106463163477504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚485262491102544 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25801161 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚488110406076928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110184001544704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3699728445952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25801159 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3720838067200 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488110406076928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚485262474063184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491810134522880 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13224731 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57404028 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110218169079808 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110218290856960 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚485409904730656 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57404028 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚488258680579072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106496343747584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13224994 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3695240960444 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491959519131136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7845096648271394e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _runtime‚2838.402866018 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491959519131136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106496325135360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57414187 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7639772 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚3199 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3721947109376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚261 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57414187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180Èç™ô&Ê îM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3700838552064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25808952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31605233 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13224731 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57404028 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110218169079808 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110218290856960 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚485409904730656 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57404028 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚488258680579072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106496343747584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13224994 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3695240960444 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491959519131136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7845096648271394e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _runtime‚2838.402866018 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491959519131136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106496325135360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57414187 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7639772 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚3199 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3721947109376 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚261 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57414187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180…¾Þ‡h2Ë - _wandbruntime‚2838  _runtime‚2838¥ä B Ì „ 4loss/FBt2/assignment_loss‚0.00042774374628838907 1loss/SRqqt1/detection_loss‚0.7237821518213541 trainer/global_step‚3199 $ _timestamp‚1.784509664828144e+09 0loss/FRt2/assignment_loss‚0.8010256833500332 1loss/SRqqt2/detection_loss‚0.6839030241580999 .loss/FBt1/detection_loss‚0.522153221508673 %loss/total_loss‚8.186633932283135 /loss/FRt1/detection_loss‚0.5117118387335822 /loss/FRt2/detection_loss‚1.5984810362696464 0loss/FBt2/detection_loss‚0.48465579139943027 2loss/SRqqt2/assignment_loss‚0.5682676092114206 epoch‚7 _runtime‚2838.403432379 _step‚262 0loss/FRt1/assignment_loss‚1.7481222411151975 2loss/SRqqt1/assignment_loss‚0.5373520162589503 2loss/FBt1/assignment_loss‚0.006751574710458298·Š©nÍ ‰ 0loss/FRt1/assignment_loss‚1.7481222411151975 2loss/SRqqt1/assignment_loss‚0.5373520162589503 2loss/FBt1/assignment_loss‚0.006751574710458298 4loss/FBt2/assignment_loss‚0.00042774374628838907 1loss/SRqqt1/detection_loss‚0.7237821518213541 trainer/global_step‚3199 $ _timestamp‚1.784509664828144e+09 0loss/FRt2/assignment_loss‚0.8010256833500332 1loss/SRqqt2/detection_loss‚0.6839030241580999 .loss/FBt1/detection_loss‚0.522153221508673 %loss/total_loss‚8.186633932283135 /loss/FRt1/detection_loss‚0.5117118387335822 /loss/FRt2/detection_loss‚1.5984810362696464 0loss/FBt2/detection_loss‚0.48465579139943027 2loss/SRqqt2/assignment_loss‚0.5682676092114206 epoch‚7 _runtime‚2838.403432379 _step‚262†n¼yRÎ ‚(:Ç ìéõÒ¸ÓåË disk.nvme1c1n1.in‚0 cpu‚25.617212833459632 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme1c1n1.out‚8637.5537109375 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚8637.5537109375" proc.memory.rssMB‚ 3001.7109375 network.sent‚ 4.89429e+06) disk./.usagePercent‚76.14465740522526 disk.md0.in‚0# disk.dm-0.out‚20179.53857421875" disk.dm-1.out‚4155.77587890625 disk.rbd0.in‚0( proc.memory.availableMB‚ 486.80859375) proc.memory.percent‚18.32098960876465 network.recv‚751227 disk.md0.out‚94.09375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625# memory_percent‚97.0287561416626 proc.cpu.threads‚17% disk./.usageGB‚1169.0108528137207% disk.nvme0c0n1.out‚15898.00390625# disk.nvme0n1.out‚15898.00390625Éìî‹žÏ ‚(:“ ìéõÒÔجÌ gpu.0.gpu‚68.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚133.855' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚44.61833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)Ë¡a2Ð - _wandbruntime‚2854  _runtime‚2854 ~Ò€Ž=Ñ ˆ{ 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6316575407981873 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48092713952064514 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20989981293678284 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3696277439594269 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7313309907913208 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48430201411247253 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8894369006156921 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 jet/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6772422790527344 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16133567690849304 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 6validation_average_jet_accuracy‚0.3572445511817932 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17459344863891602 particle/accuracy_2_of_3‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6332716941833496 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4512111246585846 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7472472190856934 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20726712048053741 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015811117365956306 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18063867092132568 $ _timestamp‚1.784509680672981e+09 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20659580826759338 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.026958027854561806 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.877173125743866 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 (particle/accuracy‚0.8452187776565552 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16147415339946747 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975974559783936 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5395833253860474 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965602159500122 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.64000004529953 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.651824414730072 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1943952888250351 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6630397439002991 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23009461164474487 *jet/accuracy_2_of_2‚0.1442248672246933 /particle/accuracy_4_of_4‚0.9504124522209167 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11128707975149155 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8942555785179138 +particle/sensitivity‚0.6243888139724731 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7298027873039246 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6157666444778442 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.789954662322998 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6403316259384155 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 +particle/specificity‚0.9056164622306824 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6790337562561035 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9546480774879456 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.39673876762390137 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996802806854248 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08329609781503677 *jet/accuracy_1_of_4‚0.9704540967941284 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2600080966949463 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.898517906665802 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.027718257158994675 /particle/accuracy_3_of_4‚0.9945282936096191 &particle/f_score‚0.634040892124176 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5623838305473328 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7203084230422974 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.809618353843689 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26392418146133423 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_2_of_4‚0.7848438024520874 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15291088819503784 *jet/accuracy_1_of_2‚0.7749506831169128 particle/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1750495880842209 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6858218312263489 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8919358849525452 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6372721195220947 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7346972227096558 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_3‚0.9080087542533875 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026679277420043945 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18653632700443268 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16241878271102905 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029676154255867004 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6762150526046753 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7374697923660278 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28827786445617676 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9917343854904175 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1943952888250351 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024377048015594482 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972619414329529 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4166666567325592 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7521466016769409 particle/accuracy_2_of_4‚1 jet/accuracy_2_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02783014066517353 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6475207209587097 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.756425678730011 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7344033122062683 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6882025599479675 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8882842063903809 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.623970627784729 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4625056982040405 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012167791835963726 particle/accuracy_1_of_4‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961074590682983 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7345592379570007 epoch‚7 trainer/global_step‚3199 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19583749771118164 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6599257588386536 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8432280421257019 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2099246233701706 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30137619376182556 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7387821674346924 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20788279175758362 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09293252229690552 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.605803906917572 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4643717408180237 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2027783840894699 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 _step‚263 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941260814666748 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.027718257158994675 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚8.203652486665012 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8883032202720642 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10082674026489258 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006742527242749929 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6317457556724548 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +jet/accuracy_1_of_1‚0.31330445408821106 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5683773756027222 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17569652199745178 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8581988215446472 *jet/accuracy_2_of_3‚0.5259262323379517 +jet/accuracy_3_of_4‚0.09580603986978531 particle/accuracy_3_of_3‚1 _runtime‚ 2854.24951354 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03569226711988449 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.789890468120575 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9959101676940918 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.730488121509552 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3702358901500702 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15774552524089813 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14615504443645477 +jet/accuracy_3_of_3‚0.07405044883489609 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983945488929749 +jet/accuracy_4_of_4‚0.01670061983168125 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.618627667427063 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6700848937034607 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47211238741874695 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6332716941833496 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8581988215446472 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028035704046487808 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9910216331481934 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.726226806640625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6145833134651184 jet/accuracy_3_of_5‚1HT~“=Ò { :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26392418146133423 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_2_of_4‚0.7848438024520874 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15291088819503784 *jet/accuracy_1_of_2‚0.7749506831169128 particle/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1750495880842209 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6858218312263489 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8919358849525452 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6372721195220947 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7346972227096558 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_3‚0.9080087542533875 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026679277420043945 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18653632700443268 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16241878271102905 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.029676154255867004 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6762150526046753 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7374697923660278 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28827786445617676 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9917343854904175 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1943952888250351 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024377048015594482 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972619414329529 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4166666567325592 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7521466016769409 particle/accuracy_2_of_4‚1 jet/accuracy_2_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02783014066517353 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6475207209587097 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.756425678730011 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7344033122062683 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6882025599479675 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8882842063903809 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.623970627784729 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4625056982040405 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012167791835963726 particle/accuracy_1_of_4‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9961074590682983 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7345592379570007 epoch‚7 trainer/global_step‚3199 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19583749771118164 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6599257588386536 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8432280421257019 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2099246233701706 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30137619376182556 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7387821674346924 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20788279175758362 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09293252229690552 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.605803906917572 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4643717408180237 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2027783840894699 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 _step‚263 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941260814666748 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.027718257158994675 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚8.203652486665012 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8883032202720642 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10082674026489258 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006742527242749929 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6317457556724548 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +jet/accuracy_1_of_1‚0.31330445408821106 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5683773756027222 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_3‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17569652199745178 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8581988215446472 *jet/accuracy_2_of_3‚0.5259262323379517 +jet/accuracy_3_of_4‚0.09580603986978531 particle/accuracy_3_of_3‚1 _runtime‚ 2854.24951354 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03569226711988449 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.789890468120575 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9959101676940918 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.730488121509552 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3702358901500702 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15774552524089813 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14615504443645477 +jet/accuracy_3_of_3‚0.07405044883489609 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983945488929749 +jet/accuracy_4_of_4‚0.01670061983168125 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.618627667427063 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6700848937034607 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47211238741874695 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6332716941833496 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8581988215446472 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028035704046487808 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9910216331481934 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.726226806640625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6145833134651184 jet/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6316575407981873 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48092713952064514 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20989981293678284 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3696277439594269 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7313309907913208 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48430201411247253 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8894369006156921 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 jet/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6772422790527344 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16133567690849304 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 6validation_average_jet_accuracy‚0.3572445511817932 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17459344863891602 particle/accuracy_2_of_3‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6332716941833496 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4512111246585846 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7472472190856934 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20726712048053741 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015811117365956306 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18063867092132568 $ _timestamp‚1.784509680672981e+09 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20659580826759338 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.026958027854561806 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.877173125743866 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 (particle/accuracy‚0.8452187776565552 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16147415339946747 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975974559783936 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5395833253860474 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965602159500122 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.64000004529953 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.651824414730072 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1943952888250351 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6630397439002991 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23009461164474487 *jet/accuracy_2_of_2‚0.1442248672246933 /particle/accuracy_4_of_4‚0.9504124522209167 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11128707975149155 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8942555785179138 +particle/sensitivity‚0.6243888139724731 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7298027873039246 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6157666444778442 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.789954662322998 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6403316259384155 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 +particle/specificity‚0.9056164622306824 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6790337562561035 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9546480774879456 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.39673876762390137 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996802806854248 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08329609781503677 *jet/accuracy_1_of_4‚0.9704540967941284 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2600080966949463 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.898517906665802 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.027718257158994675 /particle/accuracy_3_of_4‚0.9945282936096191 &particle/f_score‚0.634040892124176 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5623838305473328 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7203084230422974 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.809618353843689 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5‡dŒöUÓ ‚(:Ê ûéõÒôñÌË disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚15901.1025390625% disk.nvme0n1.out‚15901.1025390625) proc.memory.percent‚18.34561824798584) disk./.usagePercent‚76.14470386986545! disk.nvme1n1.out‚ 9663.8359375 disk.md0.out‚ 94.5703125 disk.dm-1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 738.64453125 network.sent‚ 4.956499e+06 disk.nvme1c1n1.in‚0" disk.dm-1.out‚5181.32275390625 disk.nvme0n1.in‚ 0.00390625# proc.memory.rssMB‚ 3005.74609375% disk./.usageGB‚1169.0115661621094# disk.nvme1c1n1.out‚ 9663.8359375 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0$ memory_percent‚95.49167156219482 cpu‚25.619903434393336 proc.cpu.threads‚17 network.recv‚755373 disk.nvme1n1.in‚0# disk.dm-0.out‚20182.37060546875ÛE.5Ô ‚(:“ ûéõÒæÓÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚122.887' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.96233333333333 gpu.0.smClock‚ÀCã"i1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ç65WÕ ‚(:Ì ŠêõÒˆ©ÐË disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚20183.69287109375& proc.memory.availableMB‚ 738.609375 network.sent‚ 5.002403e+06 network.recv‚759489 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚15902.5908203125' disk.nvme1c1n1.out‚9664.34716796875 disk.dm-1.out‚ 5181.42578125' disk.nvme0c0n1.out‚15902.5908203125$ memory_percent‚95.49188613891602% disk.nvme1n1.out‚9664.34716796875 disk.md0.out‚ 94.83203125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚18.34561824798584 disk.nvme0c0n1.in‚ 0.00390625# proc.memory.rssMB‚ 3005.74609375 cpu‚25.61902298298528 proc.cpu.threads‚17( disk./.usagePercent‚76.1447177844101% disk./.usageGB‚1169.0117797851562ô+ªÒÖ ‚(:’ ŠêõÒŒÄÌ gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚266.38' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.79333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r^ûV× ‚(:Ë ™êõÒƼ¶Ë disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 network.sent‚ 5.007343e+06) disk./.usagePercent‚76.14471480272196& disk.nvme0n1.out‚15905.82958984375 disk.dm-1.out‚ 5181.50390625 disk.rbd0.in‚0& proc.memory.availableMB‚ 738.609375 proc.cpu.threads‚17 network.recv‚763419$ disk./.usageGB‚1169.011734008789 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚9665.38037109375 disk.nvme1n1.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 3005.74609375) proc.memory.percent‚18.34561824798584 disk.nvme0n1.in‚ 0.00390625% disk.nvme1n1.out‚9665.38037109375 disk.md0.out‚ 95.41796875 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚ 20186.6484375( disk.nvme0c0n1.out‚15905.82958984375$ memory_percent‚95.49188613891602 cpu‚25.618327576215886 disk.md0.in‚0õ%מØ ‚(:“ ™êõÒøªñË gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚291.397' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.13233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•×â2Ù - _wandbruntime‚2897  _runtime‚2897Íbu½ÇÚ Á & lr-AdamW/pg1‚0.0005830071009840486 & lr-AdamW/pg2‚0.0005830071009840486 trainer/global_step‚3249 % _timestamp‚1.7845097235915284e+09 _runtime‚2897.167152069 _step‚264‚óOÌÛ Æ trainer/global_step‚3249 % _timestamp‚1.7845097235915284e+09 _runtime‚2897.167152069 _step‚264 & lr-AdamW/pg1‚0.0005830071009840486 & lr-AdamW/pg2‚0.0005830071009840486ˆ N«=2Ü - _wandbruntime‚2897  _runtime‚2897~O¿ºã'Ý ÝO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚497466475059712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3797319796224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚501241731982336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚498325200118758 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚494555280484368 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _step‚265 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚494555263445008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13467835 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108144737031680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108144755643904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26313619 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58465188 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13467556 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32151571 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚497466475059712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26313617 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚501242090445824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3775615386112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111942172320768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3797416676864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚498324842518124 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3769579073116 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58475333 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3249 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚501241731982336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚498325200118758 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚494555280484368 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _step‚265 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚494555263445008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13467835 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108144737031680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108144755643904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26313619 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58465188 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13467556 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32151571 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚497466475059712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26313617 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚501242090445824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3775615386112 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚111942172320768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3797416676864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚498324842518124 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3769579073116 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58475333 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3249 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32161227 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111976468858368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108177935913984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3776384060928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3776384060928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3776725492224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58492796 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5675775 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2897.780775359 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚498473739174594 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚3249 _step‚266 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚501391475054080 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚497614732522496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26321412 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3776725492224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13471820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58482637 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32161227 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58482637 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚498473381573848 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚497614749561856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7796042 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3770687461368 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5675778 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚497614749561856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.784509724204931e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13471537 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3798431876608 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚494702711151840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26321412 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 OADeviceStatsMonitor.on_train_batch_enÕµt<Žd/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32161227 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111976468858368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108177935913984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3776384060928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3776384060928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3776725492224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58492796 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5675775 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2897.780775359 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚498473739174594 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚3249 _step‚266 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚501391475054080 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚497614732522496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26321412 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3776725492224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13471820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58482637 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32161227 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58482637 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚498473381573848 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚497614749561856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7796042 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3770687461368 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5675778 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚497614749561856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 $ _timestamp‚1.784509724204931e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13471537 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3798431876608 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚494702711151840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26321412 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272Š9¯Ï{2â - _wandbruntime‚2897  _runtime‚2897|p ã € % _timestamp‚1.7845097242064683e+09 1loss/FBt1/assignment_loss‚0.06641806884264002 1loss/SRqqt2/detection_loss‚0.6463667136015228 /loss/FBt1/detection_loss‚0.4737203844549815 %loss/total_loss‚8.420697430505651 epoch‚8 _step‚267 /loss/FRt1/assignment_loss‚1.835597713241877 .loss/FRt2/detection_loss‚1.567361582029628 1loss/SRqqt1/detection_loss‚0.7739244194119881 trainer/global_step‚3249 2loss/SRqqt1/assignment_loss‚0.5671620958175019 2loss/SRqqt2/assignment_loss‚0.5656442555550474 /loss/FRt1/detection_loss‚0.4966596773868391 _runtime‚2897.781773421 0loss/FRt2/assignment_loss‚0.7561647407710552 2loss/FBt2/assignment_loss‚0.028941450527603572 /loss/FBt2/detection_loss‚0.6427363288649676‡é ä … 1loss/FBt1/assignment_loss‚0.06641806884264002 1loss/SRqqt2/detection_loss‚0.6463667136015228 /loss/FBt1/detection_loss‚0.4737203844549815 %loss/total_loss‚8.420697430505651 epoch‚8 _step‚267 /loss/FRt1/assignment_loss‚1.835597713241877 .loss/FRt2/detection_loss‚1.567361582029628 1loss/SRqqt1/detection_loss‚0.7739244194119881 trainer/global_step‚3249 2loss/SRqqt1/assignment_loss‚0.5671620958175019 2loss/SRqqt2/assignment_loss‚0.5656442555550474 /loss/FRt1/detection_loss‚0.4966596773868391 _runtime‚2897.781773421 0loss/FRt2/assignment_loss‚0.7561647407710552 2loss/FBt2/assignment_loss‚0.028941450527603572 /loss/FBt2/detection_loss‚0.6427363288649676 % _timestamp‚1.7845097242064683e+09‹xí Xå ‚(:Í ¨êõÒ©µ­Ë disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0# proc.memory.rssMB‚ 3005.74609375 proc.cpu.threads‚17 network.recv‚766539) disk./.usagePercent‚76.05765994490794& disk.nvme0n1.out‚15908.31298828125 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚9666.68798828125 disk.dm-1.out‚ 5181.67578125$ memory_percent‚95.49214839935303) proc.memory.percent‚18.34561824798584 disk.md0.in‚0 disk.dm-0.out‚20188.44140625( proc.memory.availableMB‚ 738.56640625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚15908.31298828125 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚9666.68798828125 disk.nvme1n1.in‚0 disk.md0.out‚ 96.296875 cpu‚25.617636923388392 network.sent‚ 5.010265e+06% disk./.usageGB‚1167.6752243041992 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625ÎQ/@œæ ‚(:‘ ¨êõÒƶïË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.862' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.28733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýT-¥ç ‚(:’ ·êõÒ® ³Ë gpu.0.gpu‚91.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚274.58' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.52666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕqiìTè ‚(:É ·êõÒ´”÷Ë% disk./.usageGB‚1167.6752243041992# disk.nvme1n1.out‚9667.865234375 disk.dm-0.in‚ 0.00390625 disk.dm-1.out‚ 5181.7265625 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚9667.865234375$ memory_percent‚95.49448490142822) proc.memory.percent‚18.34561824798584 disk.nvme1n1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 network.recv‚770848 disk.md0.in‚0 disk.md0.out‚ 96.70703125' disk.nvme0c0n1.out‚15912.1728515625( proc.memory.availableMB‚ 738.18359375 cpu‚25.61719666152044 proc.cpu.threads‚17 network.sent‚ 5.072717e+06) disk./.usagePercent‚76.05765994490794" disk.dm-0.out‚20192.5341796875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚15912.1728515625# proc.memory.rssMB‚ 3005.74609375•F¦ŠXé ‚(:Í ÆêõÒ£äË network.recv‚774134" disk.dm-1.out‚5181.98681640625 disk.rbd0.in‚0& disk.nvme0n1.out‚15913.45263671875$ disk.nvme1n1.out‚9668.6826171875 disk.md0.in‚0 disk.md0.out‚ 97.140625$ memory_percent‚96.99649810791016) disk./.usagePercent‚76.05765845406387 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 3005.74609375 cpu‚25.618295778904294% disk./.usageGB‚1167.6752014160156" disk.dm-0.out‚20193.4443359375( disk.nvme0c0n1.out‚15913.45263671875& disk.nvme1c1n1.out‚9668.6826171875) proc.memory.percent‚18.34561824798584 proc.cpu.threads‚17 network.sent‚ 5.079484e+06 disk.dm-0.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625% proc.memory.availableMB‚ 492.09375à“ÃA“ê ‚(:ˆ ÆêõÒÚ›£Ì gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚273.564' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.188 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í Å!2ë - _wandbruntime‚2940  _runtime‚2940|!=zÆì À & lr-AdamW/pg1‚0.0005819124961038309 & lr-AdamW/pg2‚0.0005819124961038309 trainer/global_step‚3299 $ _timestamp‚1.784509766572934e+09 _runtime‚2940.150014798 _step‚268XCqËí Å _runtime‚2940.150014798 _step‚268 & lr-AdamW/pg1‚0.0005819124961038309 & lr-AdamW/pg2‚0.0005819124961038309 trainer/global_step‚3299 $ _timestamp‚1.784509766572934e+09Œq¹nþ2î - _wandbruntime‚2940  _runtime‚2940~YEGä'ï ÞO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3853348328448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5760741 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32646366 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109803769147904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3825402847798 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508711026191360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3830843063808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚504880200166912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59339490 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508711026191360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26703269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7908487 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚113657117476352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26703267 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13669508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277Ý[Ú–ð ãO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26703269 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚505752216705766 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚508711384654848 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7908764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32646366 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13669228 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚113657001983488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚2940.150121488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚508711384654848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3853251447808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚505751859105132 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3830843063808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3853348328448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5760741 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32646366 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109803769147904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3825402847798 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508711026191360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3830843063808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚504880200166912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59339490 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508711026191360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26703269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7908487 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚113657117476352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374@³E¦Í E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26703267 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13669508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚508860769263104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5762478 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3831953162752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26711060 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32656036 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3832294594048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59367098 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7910741 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚508860769263104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚3299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508860410792448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59356939 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3831953162752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚505028457629696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13673216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚109836949417984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3854361005056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7911023 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113691415685120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59356939 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5762475 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3854466267136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32656036 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _step‚270 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3826170674776 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚505900398160856 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26711062 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0€±z²ô&ó îM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚502074227486080 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚508860769263104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5762478 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3831953162752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26711060 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32656036 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3832294594048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59367098 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7910741 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚508860769263104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚3299 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508860410792448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59356939 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3831953162752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚505028457629696 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13673216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚109836949417984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3854361005056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7911023 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113691415685120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59356939 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5762475 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3854466267136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32656036 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _step‚270 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3826170674776 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚505900398160856 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26711062 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ŽuP÷2ô - _wandbruntime‚2940  _runtime‚2940öÔ­¬õ € 0loss/FRt2/assignment_loss‚0.8513148364123948 2loss/FBt1/assignment_loss‚0.011412946948599765 /loss/FRt1/detection_loss‚0.5120882799727264 /loss/FRt2/detection_loss‚1.5444802582336572 1loss/SRqqt2/detection_loss‚0.6502809040354709 .loss/FBt2/detection_loss‚0.568974851364536 epoch‚8 _runtime‚2940.768774089 2loss/SRqqt2/assignment_loss‚0.5207876659477203 1loss/FBt2/assignment_loss‚0.17322198561363403 %loss/total_loss‚8.400790039466976 _step‚271 /loss/FRt1/assignment_loss‚1.781055401129122 trainer/global_step‚3299 % _timestamp‚1.7845097671935186e+09 2loss/SRqqt1/assignment_loss‚0.4939616668759707 1loss/SRqqt1/detection_loss‚0.7896094217354802 /loss/FBt1/detection_loss‚0.5036018211976625 :B— ö … 2loss/SRqqt1/assignment_loss‚0.4939616668759707 1loss/SRqqt1/detection_loss‚0.7896094217354802 /loss/FBt1/detection_loss‚0.5036018211976625 0loss/FRt2/assignment_loss‚0.8513148364123948 2loss/FBt1/assignment_loss‚0.011412946948599765 /loss/FRt1/detection_loss‚0.5120882799727264 /loss/FRt2/detection_loss‚1.5444802582336572 1loss/SRqqt2/detection_loss‚0.6502809040354709 .loss/FBt2/detection_loss‚0.568974851364536 epoch‚8 _runtime‚2940.768774089 2loss/SRqqt2/assignment_loss‚0.5207876659477203 1loss/FBt2/assignment_loss‚0.17322198561363403 %loss/total_loss‚8.400790039466976 _step‚271 /loss/FRt1/assignment_loss‚1.781055401129122 trainer/global_step‚3299 % _timestamp‚1.7845097671935186e+09”Ð@\÷ ‚(:Ñ ÕêõÒ•öË network.sent‚ 5.095249e+06 disk.nvme0n1.in‚ 0.00390625& disk.nvme1c1n1.out‚9669.5439453125 disk.md0.in‚0 disk.md0.out‚ 97.62890625" disk.dm-1.out‚5182.09619140625 disk.rbd0.in‚0( proc.memory.availableMB‚ 491.87109375% disk./.usageGB‚1167.6752624511719' disk.nvme0c0n1.out‚15915.3779296875% disk.nvme0n1.out‚15915.3779296875 disk.nvme1n1.in‚0) proc.memory.percent‚18.34561824798584 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 3005.74609375 cpu‚25.619631269570768 network.recv‚777627) disk./.usagePercent‚76.05766242964806 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme1n1.out‚9669.5439453125 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚20195.06298828125$ memory_percent‚96.997857093811049ß:žø ‚(:“ ÕêõÒƒÎ±Ì gpu.0.gpu‚80.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚256.633' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.54433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óÀÛ[ù ‚(:Ð äêõÒ†ØÁË' disk.nvme0c0n1.out‚15916.5986328125# disk.dm-0.out‚20195.98486328125' proc.memory.availableMB‚ 491.8515625 cpu‚25.618619139710514% disk./.usageGB‚1164.9580383300781 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚15916.5986328125' disk.nvme1c1n1.out‚9670.64697265625 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625 network.sent‚ 5.144717e+06) disk./.usagePercent‚75.88067339717168 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚9670.64697265625 disk.md0.in‚0 network.recv‚781480 disk.dm-1.out‚5182.228515625 disk.rbd0.in‚0 disk.md0.out‚ 98.22265625$ memory_percent‚96.99797630310059# proc.memory.rssMB‚ 3005.74609375) proc.memory.percent‚18.34561824798584 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.out‚0Ýâwîú ‚(:’ äêõÒö„™Ì gpu.0.gpu‚25.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚272.983' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.99433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!¢d2û - _wandbruntime‚2983  _runtime‚2983-ÃoÇü Á trainer/global_step‚3349 % _timestamp‚1.7845098102345111e+09 _runtime‚2983.810078673 _step‚272 & lr-AdamW/pg1‚0.0005808002498250306 & lr-AdamW/pg2‚0.0005808002498250306wQùÌý Æ & lr-AdamW/pg2‚0.0005808002498250306 trainer/global_step‚3349 % _timestamp‚1.7845098102345111e+09 _runtime‚2983.810078673 _step‚272 & lr-AdamW/pg1‚0.0005808002498250306½;ÏË2þ - _wandbruntime‚2983  _runtime‚2983yj$Ï‚ ÿ ßO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚516180320400384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚3349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3886412165632 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115371830582784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚273 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13871171 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27092917 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8023730 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3886412165632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3880545499932 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33131018 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚512293908234752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚509298330192208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27092919 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60213792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8024022 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚513179233292774 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚516180678863872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27092919 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13871466 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚509298347231568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3880886061206 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33120875 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚111462782651904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3886753589760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 % _timestamp‚1.7845098102354846e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚512293925274112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚512293925274112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5847444 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3909180200960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚2983.811045885 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ˆ¼óê'€äO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115371962852864 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚516180320400384 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27092917 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60223937 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚3349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3886412165632 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115371830582784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚273 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13871171 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27092917 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8023730 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3886412165632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3880545499932 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33131018 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚512293908234752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚509298330192208 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27092919 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60213792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8024022 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚513179233292774 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚516180678863872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27092919 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13871466 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚509298347231568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3880886061206 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33120875 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚111462782651904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3886753589760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 % _timestamp‚1.7845098102354846e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚512293925274112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚512293925274112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5847444 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3909180200960 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚2983.811045885 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚516180320400384 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13875119 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5849175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3887522264576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111495944309760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚512442199776256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3887522264576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60241400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27100710 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3910178811904 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27100712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚2984.430990938 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚516329705001472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 % _timestamp‚1.7845098108552587e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚306 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13875428 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚513327772348610 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60231241 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509445760859680 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33140688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8025944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚111495962921984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚512442182736896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60231241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚512442199776256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8026250 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚3349 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60241400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27100710 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5849178 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3910292462592 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115406123121664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _step‚274 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27100712 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚516330063472128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚516330063472128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚509445777899040 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33130531 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚512442182736896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33130531 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33140688 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚516330063472128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚516330063472128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚509445777899040 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33130531 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚512442182736896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33130531 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33140688 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13875119 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5849175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3887522264576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚111495944309760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚512442199776256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3887522264576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peakÓ[Ì…‚85 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60241400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27100710 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3910178811904 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27100712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚2984.430990938 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚516329705001472 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 % _timestamp‚1.7845098108552587e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚306 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13875428 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚513327772348610 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60231241 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚509445760859680 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33140688 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8025944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚111495962921984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚512442182736896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60231241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚512442199776256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8026250 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚3349’aÆë2„- _wandbruntime‚2984  _runtime‚2984D°—…‚ 0loss/FRt2/assignment_loss‚0.8717608647803737 2loss/SRqqt2/assignment_loss‚0.6739971239094581 2loss/FBt1/assignment_loss‚0.013731363491773696 2loss/FBt2/assignment_loss‚0.002852531700806629 1loss/SRqqt2/detection_loss‚0.6559274692154611 0loss/FBt2/detection_loss‚0.48518542683752736 % _timestamp‚1.7845098108562853e+09 2loss/SRqqt1/assignment_loss‚0.7362943995316613 .loss/FRt2/detection_loss‚1.586272049457107 1loss/SRqqt1/detection_loss‚0.8306098821926112 trainer/global_step‚3349 _step‚275 %loss/total_loss‚8.825406759496104 epoch‚8 0loss/FRt1/assignment_loss‚1.8280660951600491 .loss/FBt1/detection_loss‚0.625099895929452 _runtime‚2984.431533329 /loss/FRt1/detection_loss‚0.5156096572898234QHdÈ †‡ 2loss/SRqqt1/assignment_loss‚0.7362943995316613 .loss/FRt2/detection_loss‚1.586272049457107 1loss/SRqqt1/detection_loss‚0.8306098821926112 trainer/global_step‚3349 _step‚275 %loss/total_loss‚8.825406759496104 epoch‚8 0loss/FRt1/assignment_loss‚1.8280660951600491 .loss/FBt1/detection_loss‚0.625099895929452 _runtime‚2984.431533329 /loss/FRt1/detection_loss‚0.5156096572898234 0loss/FRt2/assignment_loss‚0.8717608647803737 2loss/SRqqt2/assignment_loss‚0.6739971239094581 2loss/FBt1/assignment_loss‚0.013731363491773696 2loss/FBt2/assignment_loss‚0.002852531700806629 1loss/SRqqt2/detection_loss‚0.6559274692154611 0loss/FBt2/detection_loss‚0.48518542683752736 % _timestamp‚1.7845098108562853e+09“;¤áÂZ‡‚(:Ï óêõÒçóË disk.dm-1.out‚5182.263671875$ memory_percent‚96.99978828430176' proc.memory.availableMB‚ 491.5546875) proc.memory.percent‚18.34561824798584 cpu‚25.617852681925623) disk./.usagePercent‚75.88068010596999 disk.rbd0.out‚0' disk.nvme0c0n1.out‚15921.0615234375 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 3005.74609375% disk./.usageGB‚1164.9581413269043 disk.nvme0n1.in‚ 0.00390625% disk.nvme1n1.out‚9671.75439453125 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 5.163437e+06 disk.rbd0.in‚0% disk.nvme0n1.out‚15921.0615234375' disk.nvme1c1n1.out‚9671.75439453125 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 98.8828125 network.recv‚784902 disk.nvme0c0n1.in‚ 0.00390625# disk.dm-0.out‚20200.11767578125c¹#|žˆ‚(:“ óêõÒ“¶¶Ì gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚277.864' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.62133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eðò^ž‰‚(:“ ‚ëõÒážÑË gpu.0.gpu‚40.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.654' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.55133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›2ò!EŠ‚(:º ‚ëõÒñµ©Ì proc.cpu.threads‚17 disk.nvme0n1.in‚ 0.00390625# disk.dm-0.out‚20200.60595703125 disk.dm-1.in‚0 disk.rbd0.in‚0) proc.memory.percent‚18.34561824798584 network.sent‚ 5.211991e+06 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚96.99962139129639( proc.memory.availableMB‚ 491.58203125% disk./.usageGB‚1164.9581413269043 disk.nvme0n1.out‚15922 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚9672.56005859375% disk.nvme1n1.out‚9672.56005859375 disk.dm-1.out‚5182.431640625 disk.nvme0c0n1.out‚15922# proc.memory.rssMB‚ 3005.74609375 network.recv‚788676) disk./.usagePercent‚75.88068010596999 disk.md0.in‚0 disk.md0.out‚ 99.40234375 cpu‚25.61692757346394P¯Ñ–[‹‚(:Ð ‘ëõÒÛæÐË( proc.memory.availableMB‚ 491.81640625 proc.cpu.threads‚17 network.sent‚ 5.218545e+06( disk./.usagePercent‚75.8806823422361 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚9674.44677734375 disk.md0.in‚0# disk.dm-0.out‚20209.37451171875 network.recv‚791960% disk./.usageGB‚1164.9581756591797 disk.rbd0.in‚0% disk.nvme0n1.out‚15929.6474609375 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚9674.44677734375# proc.memory.rssMB‚ 3005.74609375) proc.memory.percent‚18.34561824798584 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.md0.out‚ 99.73046875 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.dm-1.out‚5182.486328125$ memory_percent‚96.99819087982178 cpu‚25.616196278470397 disk.rbd0.out‚0' disk.nvme0c0n1.out‚15929.6474609375€ËŠ.ŸŒ‚(:” ‘ëõÒ‹é“Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚121.816' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.605333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c[uJ2- _wandbruntime‚3027  _runtime‚30276ÐroÆŽÀ $ _timestamp‚1.784509853429976e+09 _runtime‚3027.005966552 _step‚276 & lr-AdamW/pg1‚0.0005796704335851479 & lr-AdamW/pg2‚0.0005796704335851479 trainer/global_step‚3399 §‰yËÅ & lr-AdamW/pg1‚0.0005796704335851479 & lr-AdamW/pg2‚0.0005796704335851479 trainer/global_step‚3399 $ _timestamp‚1.784509853429976e+09 _runtime‚3027.005966552 _step‚276”œs†¬2- _wandbruntime‚3027  _runtime‚3027{Fgˆã'‘ÝO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113121777543680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚516669880605168 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3964981507584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚520606249879782 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚523649614609408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3942322691584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3936028713340 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61088094 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚519707650381312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚520605892279148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113121796155904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3936369274614 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚519707633341952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27482567 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27482567 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33615670 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14072012 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5934144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3964886724096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚519707633341952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117086777663488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5934141 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61098239 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚3027.010065998 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27482569 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚516669863565808 $ _timestamp‚1.784509853430706e+09 trainer/global_step‚3399 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8138139 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚523649614609408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚519707650381312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27482569 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14072283 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8137871 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚523649973072896 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3941981267456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117086664267776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224õÂç±è'’âO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚523649973072896 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3941981267456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117086664267776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113121777543680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚516669880605168 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3964981507584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚520606249879782 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚523649614609408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3942322691584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3936028713340 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61088094 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚519707650381312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚520605892279148 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113121796155904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3936369274614 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚519707633341952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27482567 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27482567 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33615670 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14072012 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5934144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3964886724096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚519707633341952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117086777663488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5934141 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61098239 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚3027.010065998 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27482569 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚516669863565808 $ _timestamp‚1.784509853430706e+09 trainer/global_step‚3399 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8138139 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚523649614609408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚519707650381312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27482569 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14072283 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8137871 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272•èæ2“- _wandbruntime‚3027  _runtime‚30271—:ï”éM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33615183 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27490362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8140415 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3937477662978 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3943432797696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚516817294233280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚523799357681152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117121079337472 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚519855924883456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3937137101592 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚516817311272640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚523799357681152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3943432797696 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5935878 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61115702 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3965993455104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14076018 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3966102911488 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3943091366400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845098540530884e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3943091366400 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117120951268864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27490362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61115702 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61105543 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113154976425984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33615183 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8140143 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚520754431334872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚519855907844096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27490360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚520754788935618 trainer/global_step‚3399 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5935875 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚519855907844096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5935875 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚519855907844096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27490362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8140415 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3937477662978 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3943432797696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚516817294233280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚523799357681152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117121079337472 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚519855924883456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3937137101592 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚516817311272640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚523799357681152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3943432797696 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5935878 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61115702 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3965993455104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14076018 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3966102911488 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3943091366400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845098540530884e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3943091366400 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117120951268864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27490362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61115702 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61105543 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113154976425984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272–t´92–- _wandbruntime‚3027  _runtime‚3027óX;—‚ /loss/FRt1/detection_loss‚0.5054548320517761 %loss/total_loss‚8.743457982645072 epoch‚8 _step‚279 3loss/FBt2/assignment_loss‚0.0074685454800197506 trainer/global_step‚3399 _runtime‚ 3027.62938404 0loss/FRt1/assignment_loss‚1.8727660060639884 1loss/SRqqt2/detection_loss‚0.6809082965759401 % _timestamp‚1.7845098540541174e+09 0loss/FRt2/assignment_loss‚0.9087282093683086 2loss/SRqqt1/assignment_loss‚0.4649772387604671 1loss/SRqqt2/assignment_loss‚0.597868735572769 2loss/FBt1/assignment_loss‚0.017875603778585582 /loss/FRt2/detection_loss‚1.7010633833404454 1loss/SRqqt1/detection_loss‚0.7898305010739664 /loss/FBt1/detection_loss‚0.5996786909163969 /loss/FBt2/detection_loss‚0.5968379396624093ÉAÛF ˜‡ epoch‚8 _step‚279 3loss/FBt2/assignment_loss‚0.0074685454800197506 trainer/global_step‚3399 _runtime‚ 3027.62938404 0loss/FRt1/assignment_loss‚1.8727660060639884 1loss/SRqqt2/detection_loss‚0.6809082965759401 % _timestamp‚1.7845098540541174e+09 0loss/FRt2/assignment_loss‚0.9087282093683086 2loss/SRqqt1/assignment_loss‚0.4649772387604671 1loss/SRqqt2/assignment_loss‚0.597868735572769 2loss/FBt1/assignment_loss‚0.017875603778585582 /loss/FRt2/detection_loss‚1.7010633833404454 1loss/SRqqt1/detection_loss‚0.7898305010739664 /loss/FBt1/detection_loss‚0.5996786909163969 /loss/FBt2/detection_loss‚0.5968379396624093 /loss/FRt1/detection_loss‚0.5054548320517761 %loss/total_loss‚8.743457982645072—H§nîV™‚(:Ë  ëõÒÐêÈË network.recv‚795107$ disk./.usageGB‚1164.958179473877 disk.md0.in‚0 disk.md0.out‚ 100.02734375 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625# proc.memory.rssMB‚ 3005.74609375 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚15930.4169921875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚5182.55517578125 disk.rbd0.in‚0$ memory_percent‚96.99845314025879) disk./.usagePercent‚75.88068259071011' proc.memory.availableMB‚ 491.7734375# disk.dm-0.out‚20209.91357421875 network.sent‚ 5.23431e+06$ disk.nvme1c1n1.out‚ 9674.94140625" disk.nvme1n1.out‚ 9674.94140625' disk.nvme0c0n1.out‚15930.4169921875) proc.memory.percent‚18.34561824798584 cpu‚25.615379004065737 proc.cpu.threads‚17ó»‹äœš‚(:‘  ëõÒ〄Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.587' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.86233333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kÃëY›‚(:Î ¯ëõÒ«ç¿Ë disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 100.3984375 cpu‚25.614733246908393 network.recv‚799399! disk.dm-0.out‚22125.126953125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚17845.80810546875 disk.md0.in‚0$ memory_percent‚96.99852466583252( proc.memory.availableMB‚ 491.76171875) proc.memory.percent‚18.34561824798584) disk./.usagePercent‚76.01096496929894" disk.dm-1.out‚5182.63720703125 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚17845.80810546875% disk.nvme1c1n1.out‚9675.662109375# disk.nvme1n1.out‚9675.662109375 proc.cpu.threads‚17 network.sent‚ 5.283731e+06 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 3005.74609375$ disk./.usageGB‚1166.958339691162>p]žœ‚(:“ ¯ëõÒ¤ž•Ì gpu.0.gpu‚91.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚283.249' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.41633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿«„X‚(:Í ¾ëõÒ·ö¬Ë$ disk./.usageGB‚1167.719783782959 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚17847.1962890625% disk.nvme0n1.out‚17847.1962890625 disk.nvme1n1.in‚0# disk.nvme1n1.out‚9676.642578125" disk.dm-1.out‚5182.74169921875 network.recv‚802788 disk.nvme0n1.in‚ 0.00390625 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚22125.91259765625 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚96.99869155883789) proc.memory.percent‚18.34561824798584 network.sent‚ 5.289603e+06 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚9676.642578125 disk.md0.in‚0 disk.md0.out‚ 101.10546875& proc.memory.availableMB‚ 491.734375 cpu‚25.613932557088102 proc.cpu.threads‚17 disk.dm-1.in‚0# proc.memory.rssMB‚ 3005.74609375) disk./.usagePercent‚76.06056236983937„V?žž‚(:“ ¾ëõÒëýøË gpu.0.gpu‚68.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.447' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.48233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)Ÿ˜2Ÿ- _wandbruntime‚3070  _runtime‚3070û‘Æ À $ _timestamp‚1.784509897004315e+09 _runtime‚3070.580718955 _step‚280 & lr-AdamW/pg1‚0.0005785231199501685 & lr-AdamW/pg2‚0.0005785231199501685 trainer/global_step‚3449l¬»Ë¡Å _step‚280 & lr-AdamW/pg1‚0.0005785231199501685 & lr-AdamW/pg2‚0.0005785231199501685 trainer/global_step‚3449 $ _timestamp‚1.784509897004315e+09 _runtime‚3070.580718955˜m ‚”2¢- _wandbruntime‚3070  _runtime‚3070ÿ†:i"£ÝO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4020840641536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34100322 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34090179 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14273652 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14273363 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚3070.581540967 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚531118908818432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚527121358449152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3997891793408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚531119267281920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚114780791047680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚528033266466790 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚524041396939408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784509897005905e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27872217 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528032908866156 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6020841 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚281 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚527121375488512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118801536905728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batcHL]bh_start/active.all.freed‚61962396 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8252808 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3997550369280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚531118908818432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8252522 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528ÁShè'¤âO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3991852488022 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34100322 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3991511926748 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6020844 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34090179 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4020745858048 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27872219 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚531119267281920 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚118801650301440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114780809659904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528032908866156 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6020841 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚281 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚527121375488512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118801536905728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61962396 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8252808 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3997550369280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚531118908818432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8252522 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4020840641536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34100322 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34090179 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14273652 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14273363 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚3070.581540967 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚531118908818432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚527121358449152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3997891793408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚531119267281920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚114780791047680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚528033266466790 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚524041396939408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784509897005905e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27872217 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚3449 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚527269649990656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14277388 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚528181447921880 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚282 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61990004 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚524188844646240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 $ _timestamp‚1.784509897627021e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34109992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61979845 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34109992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114813989929984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8255087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61990004 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0+é $ó&§íM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3999001899520 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27880012 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34099835 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚531268651890176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4021961111552 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14277388 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚528181447921880 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚282 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61990004 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚524188844646240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 $ _timestamp‚1.784509897627021e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34109992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61979845 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34109992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114813989929984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8255087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61990004 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚3071.203044461 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27880010 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚527269649990656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚527269632951296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34099835 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6022578 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6022575 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚118835951041536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚528181805522626 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3992620315000 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3998660468224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚3449 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚527269649990656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚538588203027456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6107544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚531412947352368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116439804551680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784509940346186e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚538588203027456 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚3113.921708707 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8367582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62836698 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116439823163904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4076599745536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28261867 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34574831 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4053460895232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28261867 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120516404297216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28261869 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4076700820480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4053119471104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚538588561490944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62846843 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534535100595712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34584974 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4053460895232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168ÃÀ>é'¶ãO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚538588561490944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62846843 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534535100595712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34584974 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4053460895232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34574831 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8367863 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4047335701430 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚538588561490944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14475407 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚534535100595712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚538588203027456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6107544 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚531412947352368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116439804551680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784509940346186e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚538588203027456 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚3113.921708707 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8367582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62836698 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116439823163904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4076599745536 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28261867 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34574831 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4053460895232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28261867 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120516404297216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28261869 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4076700820480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4053119471104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360{Ï›,2·- _wandbruntime‚3114  _runtime‚3114ìprîô¸èM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116473003433984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚538737946099200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62854147 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚534683358058496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6109275 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4048103528408 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚3499 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28269660 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4077712645120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62864306 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8369914 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34584487 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4054229570048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚531560378019840 ZFDeviceStatsMonitor.on_traiKãü­ú n_batch_end/requested_bytes.large_pool.freed‚531560360980480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4054571001344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14479469 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚535608464508888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚538737587628544 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14479189 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4048444089794 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34594644 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6109278¬‰ió&¹íM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚531560360980480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4054571001344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14479469 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚535608464508888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚538737587628544 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14479189 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4048444089794 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34594644 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6109278 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116473003433984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚538737946099200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62854147 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚534683358058496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6109275 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4048103528408 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚3499 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28269660 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4077712645120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62864306 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8369914 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34584487 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4054229570048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚531560378019840žù;×R2º- _wandbruntime‚3114  _runtime‚3114ÑSÓ5 »ƒ 1loss/SRqqt1/assignment_loss‚0.640177579659212 2loss/SRqqt2/assignment_loss‚0.6054069563820161 2loss/FBt1/assignment_loss‚0.006295840725381808 /loss/FRt2/detection_loss‚1.5936185612877682 1loss/SRqqt1/detection_loss‚0.8606500402587699 trainer/global_step‚3499 0loss/FRt2/assignment_loss‚0.8804730485008738 2loss/FBt2/assignment_loss‚0.004536259574114508 1loss/SRqqt2/detection_loss‚0.6805995603793962 0loss/FBt2/detection_loss‚0.46577969887959103 %loss/total_loss‚8.588116455934902 0loss/FRt1/assignment_loss‚1.8510962781451998 /loss/FRt1/detection_loss‚0.5208663149958731 % _timestamp‚1.7845099409704208e+09 /loss/FBt1/detection_loss‚0.4786163171467055 epoch‚8 _runtime‚3114.545710311 _step‚287KËL³¼ˆ /loss/FBt1/detection_loss‚0.4786163171467055 epoch‚8 _runtime‚3114.545710311 _step‚287 1loss/SRqqt1/assignment_loss‚0.640177579659212 2loss/SRqqt2/assignment_loss‚0.6054069563820161 2loss/FBt1/assignment_loss‚0.006295840725381808 /loss/FRt2/detection_loss‚1.5936185612877682 1loss/SRqqt1/detection_loss‚0.8606500402587699 trainer/global_step‚3499 0loss/FRt2/assignment_loss‚0.8804730485008738 2loss/FBt2/assignment_loss‚0.004536259574114508 1loss/SRqqt2/detection_loss‚0.6805995603793962 0loss/FBt2/detection_loss‚0.46577969887959103 %loss/total_loss‚8.588116455934902 0loss/FRt1/assignment_loss‚1.8510962781451998 /loss/FRt1/detection_loss‚0.5208663149958731 % _timestamp‚1.7845099409704208e+09Ÿª·…5X½‚(:Í úëõÒ÷ŒÑË cpu‚25.610711759547794$ disk.nvme1n1.out‚9976.7119140625$ memory_percent‚96.99914455413818) proc.memory.percent‚18.34561824798584 proc.cpu.threads‚17 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚102.6875# disk.dm-0.out‚23003.85009765625" disk.dm-1.out‚5183.05712890625 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚18428.7548828125 network.sent‚ 5.377229e+06( disk./.usagePercent‚76.0575617976733% disk./.usageGB‚1167.6737174987793 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚18428.7548828125 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 3005.74609375 network.recv‚816723 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0& disk.nvme1c1n1.out‚9976.7119140625( proc.memory.availableMB‚ 491.66015625ܵtž¾‚(:“ úëõÒéÉ™Ì gpu.0.gpu‚70.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚122.446' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.81533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T¦N¦X¿‚(:Í ‰ìõÒßç¼Ë disk.md0.out‚ 103.13671875" disk.dm-0.out‚23010.6650390625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme0n1.out‚18435.142578125# memory_percent‚96.9991683959961 disk.md0.in‚0" disk.dm-1.out‚5183.10400390625 network.recv‚820888 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚9978.15966796875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625# proc.memory.rssMB‚ 3005.74609375% proc.memory.availableMB‚ 491.65625) proc.memory.percent‚18.34561824798584 cpu‚25.609790793638957 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625 network.sent‚ 5.428587e+06) disk./.usagePercent‚76.05757720306204% disk./.usageGB‚1167.6739540100098 disk.nvme1n1.in‚0% disk.nvme1n1.out‚9978.15966796875 disk.rbd0.in‚0& disk.nvme0c0n1.out‚18435.142578125aW$À‚(:’ ‰ìõÒ¸•Ì gpu.0.gpu‚6.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.798' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.266000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãEÌæZÁ‚(:Ï ˜ìõÒëÙÙË disk.nvme0c0n1.in‚ 0.00390625# proc.memory.rssMB‚ 3005.74609375 cpu‚25.609187899328788 proc.cpu.threads‚17% disk./.usageGB‚1167.6742057800293% disk.nvme1n1.out‚9978.64111328125 disk.md0.in‚0 disk.md0.out‚103.4375 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625' proc.memory.availableMB‚ 491.6484375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚9978.64111328125 disk.nvme1n1.in‚0( disk.nvme0c0n1.out‚18436.14697265625& disk.nvme0n1.out‚18436.14697265625$ memory_percent‚96.99921607971191) proc.memory.percent‚18.34561824798584) disk./.usagePercent‚76.05759360234681 disk.dm-0.in‚ 0.00390625! disk.dm-1.out‚5183.1669921875 disk.rbd0.in‚0 network.sent‚ 5.433429e+06 network.recv‚824156" disk.dm-0.out‚23011.4541015625 disk.rbd0.out‚0™ßO°Â‚(:’ ˜ìõÒ¯Ë¥Ì gpu.0.gpu‚68.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.36' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.78666666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ʽ€A2Ã- _wandbruntime‚3157  _runtime‚3157s¨´…ÇÄÁ _runtime‚3157.569045441 _step‚288 & lr-AdamW/pg1‚0.0005761762963732553 & lr-AdamW/pg2‚0.0005761762963732553 trainer/global_step‚3549 % _timestamp‚1.7845099839923775e+09ò¶zÌÅÆ & lr-AdamW/pg1‚0.0005761762963732553 & lr-AdamW/pg2‚0.0005761762963732553 trainer/global_step‚3549 % _timestamp‚1.7845099839923775e+09 _runtime‚3157.569045441 _step‚288 V$+ž2Æ- _wandbruntime‚3157  _runtime‚3157áËä'ÇÞO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚546057855699968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35069626 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4109029997056 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8483038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚542886942040172 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4102818914838 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚3549 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35059483 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14677558 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚541948808663552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚546057497236480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4132544287232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4132449503744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚541948808663552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4102478353564 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28651519 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8483314 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28651517 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4109029997056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35069626 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35059483 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118098818055680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845099839930878e+09 _runtime‚3157.569107921 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚546057497236480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14677279 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚538784480725968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚541948825702912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4108688572928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚538784463686608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118098836667904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274âxùÇ–ÈãO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚546057497236480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4132544287232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4132449503744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚541948808663552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4102478353564 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28651519 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8483314 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28651517 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4109029997056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35069626 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35059483 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚118098818055680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845099839930878e+09 _runtime‚3157.569107921 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚546057497236480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14677279 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚^üoÇS3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚538784480725968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚541948825702912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4108688572928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚538784463686608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118098836667904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚546057855699968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35069626 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4109029997056 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8483038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚542886942040172 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4102818914838 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚3549 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35059483 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14677558 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚541948808663552¡å†É2É- _wandbruntime‚3158  _runtime‚3158¬È½üî&ÊèM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63728449 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784509984613433e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118131998325760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28659310 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _runtime‚3158.189248039 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28659312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚542097083165696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8485281 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109798671872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _step‚290 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚546206881837568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚546207240308224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122265559971328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚543035838696642 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4103927303202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63738608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35069139 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35069139 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14681256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118132016937984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚542097100205056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4133561645568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4110140103168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4109798671872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚538931911393440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚543035481095896 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6195975 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63728449ªÖÒÐó&ËíM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63728449 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63728449 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784509984613433e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118131998325760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28659310 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _runtime‚3158.189248039 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28659312 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚542097083165696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8485281 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109798671872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _step‚290 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚546206881837568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚546207240308224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122265559971328 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚543035838696642 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4103927303202 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63738608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35069139 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35069139 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14681256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118132016937984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚542097100205056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4133561645568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4110140103168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4109798671872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚538931911393440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚543035481095896 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6195975¢Ÿ|-2Ì- _wandbruntime‚3158  _runtime‚3158¢#±Í€ 2loss/FBt2/assignment_loss‚0.001375631915345681 epoch‚8 0loss/FRt2/assignment_loss‚0.8230070971451511 2loss/SRqqt1/assignment_loss‚0.5411377154052892 0loss/FRt1/detection_loss‚0.46780557110406495 /loss/FRt2/detection_loss‚1.5841603832573181 1loss/SRqqt1/detection_loss‚0.7921703708794047 1loss/SRqqt2/detection_loss‚0.6804106001347774 .loss/FBt1/detection_loss‚0.462033628281035 $loss/total_loss‚8.24834103093864 0loss/FRt1/assignment_loss‚1.8401367941132085 1loss/FBt1/assignment_loss‚0.03749895728938772 trainer/global_step‚3549 % _timestamp‚1.7845099846152415e+09 _runtime‚3158.190674862 _step‚291 2loss/SRqqt2/assignment_loss‚0.5362259361559778 .loss/FBt2/detection_loss‚0.482378345257679t\¬ñ Î… epoch‚8 0loss/FRt2/assignment_loss‚0.8230070971451511 2loss/SRqqt1/assignment_loss‚0.5411377154052892 0loss/FRt1/detection_loss‚0.46780557110406495 /loss/FRt2/detection_loss‚1.5841603832573181 1loss/SRqqt1/detection_loss‚0.7921703708794047 1loss/SRqqt2/detection_loss‚0.6804106001347774 .loss/FBt1/detection_loss‚0.462033628281035 $loss/total_loss‚8.24834103093864 0loss/FRt1/assignment_loss‚1.8401367941132085 1loss/FBt1/assignment_loss‚0.03749895728938772 trainer/global_step‚3549 % _timestamp‚1.7845099846152415e+09 _runtime‚3158.190674862 _step‚291 2loss/SRqqt2/assignment_loss‚0.5362259361559778 .loss/FBt2/detection_loss‚0.482378345257679 2loss/FBt2/assignment_loss‚0.001375631915345681£Ž ~XÏ‚(:Í §ìõÒ„¦åË disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0& disk.nvme0n1.out‚18438.42041015625( proc.memory.availableMB‚ 491.61328125 proc.cpu.threads‚17$ disk./.usageGB‚1167.674186706543 disk.md0.out‚103.875 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 3005.74609375 disk.nvme1n1.in‚0 disk.md0.in‚0" disk.dm-0.out‚23013.5048828125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚18438.42041015625' disk.nvme1c1n1.out‚9979.44580078125# memory_percent‚96.9994306564331 cpu‚25.608435732130694 network.sent‚ 5.449194e+06 network.recv‚827761) disk./.usagePercent‚76.05759235997674% disk.nvme1n1.out‚9979.44580078125! disk.dm-1.out‚5183.2490234375 disk.rbd0.out‚0) proc.memory.percent‚18.34561824798584„š>žÐ‚(:“ §ìõÒÀªÌ gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚171.854' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚57.28466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SéûYÑ‚(:Î ¶ìõÒûŠ˜Ë cpu‚25.60762411744593% disk./.usageGB‚1167.6736869812012 disk.md0.in‚0 disk.md0.out‚ 104.16796875" disk.dm-1.out‚5183.33251953125# proc.memory.rssMB‚ 3005.74609375) proc.memory.percent‚18.34561824798584 proc.cpu.threads‚17( disk./.usagePercent‚76.0575598098812& disk.nvme0n1.out‚18439.58447265625& disk.nvme1c1n1.out‚9979.9892578125 disk.nvme1n1.in‚0! disk.dm-0.out‚23014.482421875$ memory_percent‚96.99945449829102 network.sent‚ 5.498873e+06 network.recv‚832107 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚18439.58447265625$ disk.nvme1n1.out‚9979.9892578125 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 491.609375dš¨žÒ‚(:“ ¶ìõÒ¥ÎéË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚280.736' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.57866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿv†ºVÓ‚(:Ë ÅìõÒ„‡©Ë) disk./.usagePercent‚76.05759235997674$ disk./.usageGB‚1167.674186706543 disk.dm-1.in‚0" disk.dm-1.out‚5183.46142578125 disk.rbd0.in‚0& disk.nvme1c1n1.out‚9980.9482421875$ memory_percent‚96.99945449829102) proc.memory.percent‚18.34561824798584$ disk.nvme1n1.out‚9980.9482421875 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625' disk.nvme0c0n1.out‚18442.6923828125% disk.nvme0n1.out‚18442.6923828125 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 3005.74609375 cpu‚25.606963448645388 disk.nvme1n1.in‚0 disk.md0.out‚104.6875 disk.nvme0n1.in‚ 0.00390625& proc.memory.availableMB‚ 491.609375 proc.cpu.threads‚17 network.sent‚ 5.505651e+06 network.recv‚835676# disk.dm-0.out‚23017.32275390625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625ªŽuÊ‘Ô‚(:† ÅìõÒ­ÈåË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.267' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.089 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ×'ê2Õ- _wandbruntime‚3201  _runtime‚3201ÀÇÖÁ & lr-AdamW/pg1‚0.0005749769371634145 & lr-AdamW/pg2‚0.0005749769371634145 trainer/global_step‚3599 % _timestamp‚1.7845100275847168e+09 _runtime‚3201.161025989 _step‚292ë6&}Ì×Æ & lr-AdamW/pg2‚0.0005749769371634145 trainer/global_step‚3599 % _timestamp‚1.7845100275847168e+09 _runtime‚3201.161025989 _step‚292 & lr-AdamW/pg1‚0.0005749769371634145¤D#k”2Ø- _wandbruntime‚3201  _runtime‚3201rïÙÞO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4164599098880 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553526791445504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123946269043712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚3599 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4164599098880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚549362550810112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚549362533770752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6280944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4158302128246 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚553526791445504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119757850171904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚293 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29041169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚ 3201.16113416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚550314316227814 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚313 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14878357 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚553527149908992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29041167 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4188418871808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29041167 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6280941 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35544135 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123946143065088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚553527149908992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4164257674752 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚549362550810112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4164257674752 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14878670 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4157961566972 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29041169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8597416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚546155997060208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4188311505408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚549362533770752 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119757831559680 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64585302 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚550313958627180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534| ™xé'ÚãO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553526791445504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123946269043712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚3599 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4164599098880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚549362550810112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚549362533770752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6280944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4158302128246 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚553526791445504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119757850171904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚293 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29041169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚ 3201.16113416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚550314316227814 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚313 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14878357 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚553527149908992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29041167 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4188418871808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29041167 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6280941 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35544135 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123946143065088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚553527149908992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4164257674752 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚549362550810112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4164257674752 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14878670 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4157961566972 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29041169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8597416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚546155997060208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4188311505408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚549362533770752 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119757831559680 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64585302 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚550313958627180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4164599098880 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29048962 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64602751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚549510808272896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64612910 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4159410516610 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4165367773696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚546303427727680 $ _timestamp‚1.784510028207505e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚549510825312256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4189537678848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14882403 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚546303444767040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚3599 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚550462855283650 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8599728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553676534517248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _step‚294 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚549510808272896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14882687 _runtime‚3201.783299483 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35563948 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64612910 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553676176046592zv+‹þ ÝíM EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64602751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚549510808272896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64612910 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 \JDeviceStatsMonitor.onD\Nõ_train_batch_end/requested_bytes.small_pool.allocated‚ 4159410516610 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4165367773696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚546303427727680 $ _timestamp‚1.784510028207505e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚549510825312256 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4189537678848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14882403 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚546303444767040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚3599 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚550462855283650 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8599728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553676534517248 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _step‚294 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚549510808272896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14882687 _runtime‚3201.783299483 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35563948 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64612910 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553676176046592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4165709204992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29048962 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6282678 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚123980568120832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29048962 P&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6647785902023315 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9768972992897034 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9930570721626282 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1904444843530655 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8887755274772644 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7622628211975098 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7126873731613159 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08294451981782913 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.804816722869873 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01110590249300003 +jet/accuracy_1_of_1‚0.28865352272987366 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6640625 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2786957025527954 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9806195497512817 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14375273883342743 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9961031675338745 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6166470050811768 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19013674557209015 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6470719575881958 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6942563056945801 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7553747296333313 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020060406997799873 +particle/sensitivity‚0.6114728450775146 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1576690971851349 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17603467404842377 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1468116194009781 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7576594352722168 *jet/accuracy_2_of_2‚0.1385866403579712 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6634729504585266 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7363952398300171 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08267013728618622 particle/accuracy_4_of_5‚1 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6343997120857239 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940552711486816 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9647471308708191 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8844671845436096 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5766661167144775 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9137980341911316 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5491881370544434 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5773851871490479 jet/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23806644976139069 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6338893175125122 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16278411448001862 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6018017530441284 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8904195427894592 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.560416579246521 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02106049284338951 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9858006238937378 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4690101742744446 *jet/accuracy_2_of_3‚0.5133084058761597 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992343008518219 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_3_of_3‚0.07800228893756866 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9937809705734253 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 particle/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3535788655281067 *jet/accuracy_1_of_2‚0.7686318159103394 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.3381022810935974 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17682510614395142 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9823736548423767 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5463847517967224 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8614264130592346 particle/accuracy_1_of_2‚1 epoch‚8 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8753212690353394 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9068333506584167 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08168691396713257 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21235613524913788 particle/accuracy_3_of_3‚1 trainer/global_step‚3599 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028804458677768707 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1774076372385025 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5163824558258057 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6921376585960388 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6955292820930481 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.804816722869873 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6076840162277222 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44612017273902893 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1339958906173706 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7508587837219238 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9391363859176636 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5739085078239441 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6937682628631592 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_1_of_3‚0.9073570966720581 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08165734261274338 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6685730218887329 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19032765924930573 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.720597505569458 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7845100447382329e+09 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4751259386539459 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16068656742572784 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5683770179748535 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9896385669708252 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5785735845565796 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3246462941169739 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03851056843996048 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1924888789653778 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8089569807052612 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6714004278182983 .particle/accuracy_4_of_4‚0.933090090751648 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.041291993111371994 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21161435544490814 +jet/accuracy_4_of_4‚0.01670061983168125 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 (particle/accuracy‚0.8488730192184448 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6265134811401367 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.624142587184906 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5709894895553589 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07845707982778549 'jet/accuracy_1_of_4‚0.9664346575737 particle/accuracy_1_of_1‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33591312170028687 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5709894895553589 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.582416832447052 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8785746693611145 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8832578063011169 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34062039852142334 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19265082478523254 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15133792161941528 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6998991966247559 particle/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6075927019119263 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6940642595291138 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.921296238899231 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983664155006409 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16402415931224823 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3640274703502655 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24497829377651215 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19332706928253174 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941686987876892 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17603467404842377 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6606269478797913 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3750000298023224 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07845707982778549 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_1_of_5‚1 _runtime‚3218.318043156 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16934408247470856 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9094283580780029 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9860069751739502 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_3‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17426659166812897 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7942764759063721 _step‚296 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975493550300598 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13598217070102692 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8611111044883728 'particle/f_score‚0.6347311735153198 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_5‚0 )loss/val_total_loss‚8.221251941199622 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8611111044883728%K*Êå’{ 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3535788655281067 *jet/accuracy_1_of_2‚0.7686318159103394 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.3381022810935974 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17682510614395142 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9823736548423767 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5463847517967224 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_2_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8614264130592346 particle/accuracy_1_of_2‚1 epoch‚8 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8753212690353394 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9068333506584167 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08168691396713257 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21235613524913788 particle/accuracy_3_of_3‚1 trainer/global_step‚3599 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028804458677768707 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1774076372385025 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5163824558258057 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6921376585960388 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6955292820930481 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.804816722869873 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6076840162277222 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44612017273902893 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1339958906173706 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7508587837219238 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9391363859176636 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5739085078239441 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6937682628631592 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_1_of_3‚0.9073570966720581 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08165734261274338 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6685730218887329 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19032765924930573 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.720597505569458 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7845100447382329e+09 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4751259386539459 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16068656742572784 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5683770179748535 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9896385669708252 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5785735845565796 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3246462941169739 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03851056843996048 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1924888789653778 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8089569807052612 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6714004278182983 .particle/accuracy_4_of_4‚0.933090090751648 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.041291993111371994 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21161435544490814 +jet/accuracy_4_of_4‚0.01670061983168125 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 (particle/accuracy‚0.8488730192184448 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6265134811401367 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.624142587184906 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5709894895553589 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07845707982778549 'jet/accuracy_1_of_4‚0.9664346575737 particle/accuracy_1_of_1‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33591312170028687 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5709894895553589 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.582416832447052 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8785746693611145 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8832578063011169 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34062039852142334 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19265082478523254 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15133792161941528 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6998991966247559 particle/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6075927019119263 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6940642595291138 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.921296238899231 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983664155006409 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16402415931224823 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3640274703502655 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt1SRqq“Þ—Î"t2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24497829377651215 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19332706928253174 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941686987876892 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17603467404842377 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6606269478797913 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3750000298023224 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07845707982778549 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_1_of_5‚1 _runtime‚3218.318043156 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16934408247470856 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9094283580780029 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9860069751739502 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_3‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17426659166812897 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7942764759063721 _step‚296 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975493550300598 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13598217070102692 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8611111044883728 'particle/f_score‚0.6347311735153198 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_5‚0 )loss/val_total_loss‚8.221251941199622 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8611111044883728 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17856267094612122 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4027777910232544 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9953252077102661 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6914652585983276 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 particle/accuracy_2_of_4‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.661414623260498 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15980450809001923 +jet/accuracy_3_of_4‚0.09602249413728714 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6647785902023315 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9768972992897034 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9930570721626282 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1904444843530655 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8887755274772644 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7622628211975098 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7126873731613159 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08294451981782913 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.804816722869873 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01110590249300003 +jet/accuracy_1_of_1‚0.28865352272987366 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6640625 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2786957025527954 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9806195497512817 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14375273883342743 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9961031675338745 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6166470050811768 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19013674557209015 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6470719575881958 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6942563056945801 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7553747296333313 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020060406997799873 +particle/sensitivity‚0.6114728450775146 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1576690971851349 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17603467404842377 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1468116194009781 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7576594352722168 *jet/accuracy_2_of_2‚0.1385866403579712 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6634729504585266 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7363952398300171 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08267013728618622 particle/accuracy_4_of_5‚1 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6343997120857239 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940552711486816 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9647471308708191 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8844671845436096 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5766661167144775 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9137980341911316 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5491881370544434 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5773851871490479 jet/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23806644976139069 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6338893175125122 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16278411448001862 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6018017530441284 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8904195427894592 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.560416579246521 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02106049284338951 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9858006238937378 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4690101742744446 *jet/accuracy_2_of_3‚0.5133084058761597 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992343008518219 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_3_of_3‚0.07800228893756866 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9937809705734253 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 particle/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697¨”ß=¿\æ‚(:Ñ ãìõÒèøÅË( proc.memory.availableMB‚ 739.19140625( disk.nvme0c0n1.out‚18451.78564453125 disk.nvme0n1.in‚ 0.00390625$ disk.nvme1n1.out‚10229.017578125 disk.dm-1.in‚0 disk.rbd0.in‚0 cpu‚25.61993048534826 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625# proc.memory.rssMB‚ 3011.16015625* proc.memory.percent‚18.378663063049316 network.recv‚843106) disk./.usagePercent‚76.00455980622367% disk./.usageGB‚1166.8600044250488& disk.nvme0n1.out‚18451.78564453125 disk.md0.in‚0 disk.md0.out‚ 106.16796875" disk.dm-1.out‚5429.03857421875 disk.rbd0.out‚0 network.sent‚ 5.570879e+06 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚10229.017578125 disk.nvme1n1.in‚0! disk.dm-0.out‚23025.822265625# memory_percent‚95.4883337020874ã}s3ç‚(:’ ãìõÒÐÔ¹Ì gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚272.96' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.98666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]X~3Zè‚(:Ï òìõÒ›” Ë network.recv‚846881( disk.nvme0c0n1.out‚18452.68994140625 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625 proc.cpu.threads‚17% disk./.usageGB‚1166.8600578308105 disk.dm-1.in‚0" disk.dm-1.out‚5674.51904296875 disk.rbd0.out‚0& disk.nvme0n1.out‚18452.68994140625 disk.md0.in‚0* proc.memory.percent‚18.378663063049316 network.sent‚ 5.628781e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625$ memory_percent‚96.99068069458008& proc.memory.availableMB‚ 493.046875# proc.memory.rssMB‚ 3011.16015625) disk./.usagePercent‚76.00456328485983 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚10475.07421875# disk.nvme1n1.out‚10475.07421875 disk.md0.out‚ 106.58203125! disk.dm-0.out‚23026.400390625 cpu‚25.620227457984726|Gñ¤žé‚(:“ òìõÒÃÌìË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚136.972' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚45.65733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^p Û[ê‚(:Ð íõÒÀÏÀË disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625$ memory_percent‚96.99070453643799 proc.cpu.threads‚17 network.sent‚ 5.634426e+06 disk.dm-1.out‚5674.783203125( disk.nvme0c0n1.out‚18453.97314453125& disk.nvme0n1.out‚18453.97314453125( disk.nvme1c1n1.out‚10475.89697265625& disk.nvme1n1.out‚10475.89697265625 disk.md0.in‚0 cpu‚25.622117529227893 network.recv‚850602) disk./.usagePercent‚76.00451035989533 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 106.9609375 disk.dm-0.out‚ 23027.421875( proc.memory.availableMB‚ 493.04296875$ disk./.usageGB‚1166.859245300293 disk.dm-1.in‚0# proc.memory.rssMB‚ 3011.16015625* proc.memory.percent‚18.378663063049316äØ»˜œë‚(:‘ íõÒÃèùË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.721' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.57366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íBŽ22ì- _wandbruntime‚3263  _runtime‚3263jz“ÆíÀ & lr-AdamW/pg1‚0.0005737603820129817 & lr-AdamW/pg2‚0.0005737603820129817 trainer/global_step‚3649 $ _timestamp‚1.784510090125268e+09 _runtime‚3263.700969825 _step‚297ÌùïÔËîÅ trainer/global_step‚3649 $ _timestamp‚1.784510090125268e+09 _runtime‚3263.700969825 _step‚297 & lr-AdamW/pg1‚0.0005737603820129817 & lr-AdamW/pg2‚0.0005737603820129817©™5x2ï- _wandbruntime‚3263  _runtime‚3263Ívžå'ðßO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚559682892671948 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚125704354092544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚555448786442032 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4233748629282 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚121439423726080 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36120429 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚558718619792896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8753092 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15124852 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6371760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚559682535071314 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65674056 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4240486039040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4234089190556 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4240144614912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562959105831936 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845100901264544e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562958747368448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6371757 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36120429 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36110286 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚558718619792896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125704230211072 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65663911 trainer/global_step‚3649 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562959105831936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4240486039040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29553625 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36110286 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29553627 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚3263.702136906 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚558718602753536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 _step‚298 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4240144614912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4264911754240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4264806484992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555448803481392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8752818BîÌý$ñäO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8753092 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15124852 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6371760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚559682535071314 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65674056 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4240486039040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4234089190556 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4240144614912 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562959105831936 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845100901264544e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562958747368448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6371757 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36120429 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36110286 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚558718619792896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125704230211072 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65663911 trainer/global_step‚3649 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562959105831936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4240486039040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29553625 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36110286 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29553627 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚3263.702136906 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚558718602753536 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 _step‚298 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4240144614912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4264911754240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4264806484992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555448803481392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8752818 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚559682892671948 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚125704354092544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚555448786442032 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4233748629282 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚121439423726080 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36120429 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145  ³ÕG?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚558718619792896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184ªV.Äá2ò- _wandbruntime‚3264  _runtime‚3264àñ߇î&óèM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65681360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36130099 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4241596145152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15128614 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845100907569556e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65691519 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4241254713856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4265924365312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125738643847168 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29561420 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚558866894295040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4234857017534 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555596234148864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65681360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15128886 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4241596145152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4241254713856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29561418 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚121472603996160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _runtime‚3264.332732819 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29561420 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚563108131969536 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29561418 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360“œDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4241596145152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15128614 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845100907569556e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65691519 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4241254713856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4265924365312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125738643847168 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29561420 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚558866894295040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4234857017534 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555596234148864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65681360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15128886 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4241596145152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4241254713856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29561418 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚121472603996160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _runtime‚3264.332732819 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29561420«­ï®û2õ- _wandbruntime‚3264  _runtime‚3264•&R'ö 0loss/FRt2/assignment_loss‚0.8013472063963347 2loss/SRqqt1/assignment_loss‚0.6113146495127432 1loss/FBt1/assignment_loss‚0.01391784977301506 /loss/FRt1/detection_loss‚0.5254333635724129 0loss/FBt1/detection_loss‚0.44601532673808714 %loss/total_loss‚8.362559527405228 epoch‚9 trainer/global_step‚3649 0loss/FRt1/assignment_loss‚1.7809062868929246 2loss/SRqqt2/assignment_loss‚0.6152130893634372 /loss/FRt2/detection_loss‚1.4928412077589028 1loss/SRqqt2/detection_loss‚0.7034750682679418 _step‚300 1loss/FBt2/assignment_loss‚0.01616215351388082 /loss/FBt2/detection_loss‚0.5153558746512447 % _timestamp‚1.7845100907580447e+09 1loss/SRqqt1/detection_loss‚0.8405774509643027 _runtime‚ 3264.33334956»€“à ÷† 0loss/FRt1/assignment_loss‚1.7809062868929246 2loss/SRqqt2/assignment_loss‚0.6152130893634372 /loss/FRt2/detection_loss‚1.4928412077589028 1loss/SRqqt2/detection_loss‚0.7034750682679418 _step‚300 1loss/FBt2/assignment_loss‚0.01616215351388082 /loss/FBt2/detection_loss‚0.5153558746512447 % _timestamp‚1.7845100907580447e+09 1loss/SRqqt1/detection_loss‚0.8405774509643027 _runtime‚ 3264.33334956 0loss/FRt2/assignment_loss‚0.8013472063963347 2loss/SRqqt1/assignment_loss‚0.6113146495127432 1loss/FBt1/assignment_loss‚0.01391784977301506 /loss/FRt1/detection_loss‚0.5254333635724129 0loss/FBt1/detection_loss‚0.44601532673808714 %loss/total_loss‚8.362559527405228 epoch‚9 trainer/global_step‚3649¬ÎC&Å_ø‚(:Ô íõÒ«ìÖË disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚18.378663063049316 proc.cpu.threads‚17 network.sent‚ 5.650389e+06 disk.dm-0.out‚23028.23046875 disk.dm-1.in‚0 disk.dm-1.out‚5674.857421875& disk.nvme0n1.out‚18455.07177734375 disk.nvme1n1.in‚0) disk./.usagePercent‚75.98903241675444 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚18455.07177734375 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚10476.42529296875( proc.memory.availableMB‚ 493.01171875 cpu‚25.623839547607325 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚10476.42529296875 disk.md0.in‚0 disk.md0.out‚ 107.30078125$ memory_percent‚96.99089527130127# proc.memory.rssMB‚ 3011.16015625 network.recv‚853817% disk./.usageGB‚1166.6216201782227RúYƒžù‚(:“ íõÒíðŽÌ gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.021' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.67366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00 á [ú‚(:Ð ŸíõÒ¡èÆË* proc.memory.percent‚18.378663063049316 proc.cpu.threads‚17 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625! disk.dm-1.out‚5675.0537109375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0c0n1.out‚18459.548828125 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0# disk.dm-0.out‚23032.33544921875$ disk.nvme0n1.out‚18459.548828125& disk.nvme1n1.out‚10477.38427734375$ memory_percent‚96.99091911315918# proc.memory.rssMB‚ 3011.16015625 cpu‚25.625929426898338 network.sent‚ 5.701695e+06 network.recv‚858188) disk./.usagePercent‚75.98902993201433# disk./.usageGB‚1166.62158203125( disk.nvme1c1n1.out‚10477.38427734375 disk.md0.out‚ 107.8359375 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0' proc.memory.availableMB‚ 493.0078125Üàûžû‚(:“ ŸíõÒ ×„Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚267.041' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.01366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Áž]ü‚(:Ò ®íõÒ¡é«Ë% disk.nvme0n1.out‚18460.6845703125 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 108.234375 disk.dm-0.in‚ 0.00390625! disk.dm-1.out‚5677.9833984375 disk.rbd0.in‚0$ memory_percent‚96.99158668518066( disk.nvme1c1n1.out‚10480.89990234375 disk.nvme0c0n1.in‚ 0.00390625* proc.memory.percent‚18.378663063049316 proc.cpu.threads‚17 network.recv‚861760) disk./.usagePercent‚75.98902968354032% disk./.usageGB‚1166.6215782165527 disk.nvme0n1.in‚ 0.00390625# disk.dm-0.out‚23033.18994140625 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚18460.6845703125 network.sent‚ 5.706698e+06& disk.nvme1n1.out‚10480.89990234375 disk.md0.in‚0' proc.memory.availableMB‚ 492.8984375# proc.memory.rssMB‚ 3011.16015625 cpu‚25.62687097980632º\)Wžý‚(:“ ®íõÒßàùË gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚275.257' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.75233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°(Ý|2þ- _wandbruntime‚3308  _runtime‚3308æN}zÅÿ¿ & lr-AdamW/pg2‚0.0005725267090590209 trainer/global_step‚3699 $ _timestamp‚1.784510134810233e+09 _runtime‚ 3308.38596542 _step‚301 & lr-AdamW/pg1‚0.0005725267090590209"&RÊ€Ä & lr-AdamW/pg1‚0.0005725267090590209 & lr-AdamW/pg2‚0.0005725267090590209 trainer/global_step‚3699 $ _timestamp‚1.784510134810233e+09 _runtime‚ 3308.38596542 _step‚301­xq<‡2- _wandbruntime‚3308  _runtime‚3308í£Çq‚ÞO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29943277 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29943277 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4289572403964 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4289231842690 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚567109551658322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36594938 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚570428041577472 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8867572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127419098189312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚566132327860736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123098455842304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66538213 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4295713716736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66548358 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36605081 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4320660959232 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4296055140864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8867280 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚562820336854992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15326032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚3308.389223085 _step‚302 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚570428041577472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.\*b«son_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15325737 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚566132327860736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚570428400040960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36594938 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123098437230080 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚562820319815632 $ _timestamp‚1.784510134813507e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6458457 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6458460 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚570428400040960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚566132344900096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29943275 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚566132344900096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4320764131328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240x¬‚jé'ƒãO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29943275 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4296055140864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4295713716736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127419219973632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚567109909258956 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚3699 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚570428041577472 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8867572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127419098189312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚566132327860736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123098455842304 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66538213 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4295713716736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66548358 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36605081 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4320660959232 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4296055140864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8867280 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚562820336854992 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15326032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚3308.389223085 _step‚302 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚570428041577472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15325737 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚566132327860736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚570428400040960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36594938 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123098437230080 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚562820319815632 $ _timestamp‚1.784510134813507e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6458457 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6458460 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚570428400040960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚566132344900096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29943275 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚566132344900096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4320764131328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29943277 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29943277 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4289572403964 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4289231842690 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚567109551658322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36594938 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29951068 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8869503 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _step‚303 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36614751 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66555662 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4297165246976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29951070 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚566280602362880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6460191 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4290340230942 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚306 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚303 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15329694 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36614751 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6460194 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127453519386624 _runtime‚3309.010628587 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8869806 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4297165246976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚562967767522464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚570577426178560 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127453391318016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚567258448314792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66555662 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66565821 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4296823815680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚562967750483104 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66565821 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4321883274240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0Û`†îM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36614751 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66555662 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4297165246976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29951070 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚566280602362880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6460191 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4290340230942 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚306 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚303 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15329694 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36614751 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6460194 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127453519386624 _runtime‚3309.010628587 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8869806 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4297165246976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚562967767522464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚570577426178560 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127453391318016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚567258448314792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66555662 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66565821 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4296823815680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚562967750483104 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66565821 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4321883274240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_„#”batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29951068 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8869503 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _step‚303 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452¯@”n2‡- _wandbruntime‚3309  _runtime‚3309䩈 _runtime‚3309.011150767 0loss/FRt2/assignment_loss‚0.8136731167228854 1loss/SRqqt1/assignment_loss‚0.647928530384011 2loss/FBt1/assignment_loss‚0.002670977607657827 1loss/SRqqt2/detection_loss‚0.7264508781624041 % _timestamp‚1.7845101354358962e+09 2loss/FBt2/assignment_loss‚0.005950467211126427 /loss/FBt1/detection_loss‚0.4073924537361852 %loss/total_loss‚8.664669196994282 epoch‚9 trainer/global_step‚3699 /loss/FRt1/assignment_loss‚1.851435809440556 2loss/SRqqt2/assignment_loss‚0.5748186143080388 /loss/FRt1/detection_loss‚0.5198626202650086 /loss/FBt2/detection_loss‚0.6758676430742655 _step‚304 /loss/FRt2/detection_loss‚1.6264899075390293 1loss/SRqqt1/detection_loss‚0.8121281785431143éÑÆ ‰† /loss/FBt1/detection_loss‚0.4073924537361852 %loss/total_loss‚8.664669196994282 epoch‚9 trainer/global_step‚3699 /loss/FRt1/assignment_loss‚1.851435809440556 2loss/SRqqt2/assignment_loss‚0.5748186143080388 /loss/FRt1/detection_loss‚0.5198626202650086 /loss/FBt2/detection_loss‚0.6758676430742655 _step‚304 /loss/FRt2/detection_loss‚1.6264899075390293 1loss/SRqqt1/detection_loss‚0.8121281785431143 _runtime‚3309.011150767 0loss/FRt2/assignment_loss‚0.8136731167228854 1loss/SRqqt1/assignment_loss‚0.647928530384011 2loss/FBt1/assignment_loss‚0.002670977607657827 1loss/SRqqt2/detection_loss‚0.7264508781624041 % _timestamp‚1.7845101354358962e+09 2loss/FBt2/assignment_loss‚0.005950467211126427°ÙûnýYŠ‚(:Î ½íõÒ­ó€Ì# disk.nvme0n1.out‚18467.95703125 disk.md0.in‚0" disk.dm-0.out‚23061.2548828125" disk.dm-1.out‚5678.20263671875% disk.nvme0c0n1.out‚18467.95703125% proc.memory.availableMB‚ 492.59375 cpu‚25.626739266935132 network.recv‚864988 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.md0.out‚ 108.7578125 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625* proc.memory.percent‚18.378663063049316 network.sent‚ 5.722505e+06) disk./.usagePercent‚75.99321274352839% disk./.usageGB‚1166.6857986450195 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17( disk.nvme1c1n1.out‚10503.03759765625 disk.nvme1n1.in‚0& disk.nvme1n1.out‚10503.03759765625$ memory_percent‚96.99344635009766# proc.memory.rssMB‚ 3011.16015625ª-tž‹‚(:“ ½íõÒѵÌ gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚261.189' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.06300000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýê1¦SŒ‚(:È ÌíõҀǻË" disk.dm-1.out‚5678.29248046875 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚10539.02197265625 proc.cpu.threads‚17) disk./.usagePercent‚75.81629228987924 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.out‚ 18469.21875( disk.nvme1c1n1.out‚10539.02197265625 disk.dm-0.in‚ 0.00390625% proc.memory.availableMB‚ 492.09375# proc.memory.rssMB‚ 3011.16015625 network.sent‚ 5.773768e+06 network.recv‚868772 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 109.10546875* proc.memory.percent‚18.378663063049316$ disk./.usageGB‚1163.969627380371" disk.nvme0c0n1.out‚ 18469.21875" disk.dm-0.out‚23097.6689453125$ memory_percent‚96.99649810791016 cpu‚25.627972893735503Õó± ž‚(:“ ÌíõÒÒô¸Ë gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.474' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.49133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©ª¹(žŽ‚(:“ ÛíõÒì”Ë gpu.0.gpu‚54.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.558' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.18600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à = _‚(:Ô ÛíõÒæÒ©Ë* proc.memory.percent‚18.378663063049316 proc.cpu.threads‚17 network.recv‚872397 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625# proc.memory.rssMB‚ 3011.16015625 network.sent‚ 5.7786e+06) disk./.usagePercent‚75.81215470063533& disk.nvme1n1.out‚10572.72509765625" disk.dm-0.out‚23291.9521484375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚18630.70361328125( disk.nvme1c1n1.out‚10572.72509765625$ memory_percent‚96.99552059173584 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625! disk.dm-1.out‚5678.4892578125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚18630.70361328125 cpu‚25.629932895865533$ disk./.usageGB‚1163.906105041504 disk.md0.out‚ 109.42578125 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 492.25390625&¤ïá2- _wandbruntime‚3351  _runtime‚3351¡›ÓÇ‘Á _runtime‚3351.890443376 _step‚305 & lr-AdamW/pg1‚0.0005712759975380408 & lr-AdamW/pg2‚0.0005712759975380408 trainer/global_step‚3749 % _timestamp‚1.7845101783148777e+09z¯`CÌ’Æ & lr-AdamW/pg1‚0.0005712759975380408 & lr-AdamW/pg2‚0.0005712759975380408 trainer/global_step‚3749 % _timestamp‚1.7845101783148777e+09 _runtime‚3351.890443376 _step‚305±RÓv«2“- _wandbruntime‚3351  _runtime‚3351f¿¥å'”ßO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30332925 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚574536925845964 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚577897335786496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚570191853189232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚577897335786496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4351282818560 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67422660 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4376555922944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚3749 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚577897694249984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15528059 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37089733 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8982622 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67412515 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚573546052967936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚573546070007296 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4344715056098 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573546052967936 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚570191870228592 _runtime‚3351.891421608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37079590 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚574536568245330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124757469346304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124757450734080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4376452750848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8982899 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30332925 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚574536925845964 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚577897335786496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚570191853189232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚577897335786496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4351282818560 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67422660 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4376555922944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚3749 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚577897694249984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15528059 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37089733 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8982622 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67412515 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚573546052967936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚573546070007296 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4344715056098 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573546052967936 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚570191870228592 _runtime‚3351.891421608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37079590 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚574536568245330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124757469346304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124757450734080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4376452750848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8982899 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4377563586048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 % _timestamp‚1.7845101789377935e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚578047078858240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚573694344509440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8985178 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15531788 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ ŸÉú#561386496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67440123 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37099403 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚574685464901800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4352734348800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6546894 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30340718 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37099403 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚578046720387584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4346164005736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚570339300896064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360y–öüô&˜îM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352392917504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 _runtime‚3352.513582026 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚578047078858240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚574685107301054 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚573694344509440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚3749 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67429964 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129168194590208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15531788 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67440123 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37099403 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚574685464901800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4352734348800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6546894 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30340718 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37099403 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚578046720387584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4346164005736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚570339300896064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6546891 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129168324755968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4377563586048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 % _timestamp‚1.7845101789377935e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚578047078858240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚573694344509440 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8985178 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9097943 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9097656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4400538830780 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37574385 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15729513 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126416464238080 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30722577 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37574385 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚581963942432972 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚585366629995520 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚577563403602192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6631860 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4432267097600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37564242 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68286817 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126416482850304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6631857 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚580959795114496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4406851920384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 % _timestamp‚1.7845102218602483e+09 _step‚310 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚585366629995520 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚577563403602192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_trN¦› }$ain_batch_start/inactive_split.large_pool.allocated‚6631860 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4432267097600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37564242 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68286817 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126416482850304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6631857 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚580959795114496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4406851920384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 % _timestamp‚1.7845102218602483e+09 _step‚310 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9097943 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9097656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4400538830780 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37574385 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15729513 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126416464238080 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30722577 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37574385 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224¶Ú'Ö2¨- _wandbruntime‚3396  _runtime‚3396}vì&©æM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68304266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130883033680384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4401647219144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68304266 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚582112481488808 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37584055 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚581108052577280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37584055 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4407962019328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577710834269664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68314425 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15733850 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126449663120384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3396.056937056 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30730368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚585516373067264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68314425 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37573898 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4433389172224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577710817230304 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚581108069616640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6633594 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37573898 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4401306657758 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4407962019328 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4408303450624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126449644508160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30730370 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15733554q´yñ&ªëM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4408303450624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126449644508160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30730370 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15733554 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68304266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130883033680384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4401647219144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68304266 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚582112481488808 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37584055 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚581108052577280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37584055 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4407962019328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577710834269664 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68314425 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15733850 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126449663120384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3396.056937056 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30730368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚585516373067264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68314425 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37573898 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4433389172224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577710817230304 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚581108069616640 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6633594 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37573898 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4401306657758 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4407962019328·{­¾œ2«- _wandbruntime‚3396  _runtime‚3396‘üì€ % _timestamp‚1.7845102224822628e+09 _step‚312 /loss/FBt2/detection_loss‚0.9591722284796936 0loss/FRt1/assignment_loss‚1.7112832222897758 0loss/FRt2/assignment_loss‚0.7816256525510564 1loss/FBt1/assignment_loss‚0.03730960677238178 1loss/SRqqt1/detection_loss‚0.7373545944736507 /loss/FBt1/detection_loss‚0.5525360054228443 .loss/FRt1/detection_loss‚0.522709178885194 /loss/FRt2/detection_loss‚1.5380702190486681 epoch‚9 trainer/global_step‚3799 _runtime‚3396.057555967 2loss/SRqqt2/assignment_loss‚0.5710405533382902 1loss/FBt2/assignment_loss‚0.06749443473431267 1loss/SRqqt2/detection_loss‚0.7294585594013333 2loss/SRqqt1/assignment_loss‚0.5153373306265032 %loss/total_loss‚8.723391586023704ƒú§ ­… /loss/FBt1/detection_loss‚0.5525360054228443 .loss/FRt1/detection_loss‚0.522709178885194 /loss/FRt2/detection_loss‚1.5380702190486681 epoch‚9 trainer/global_step‚3799 _runtime‚3396.057555967 2loss/SRqqt2/assignment_loss‚0.5710405533382902 1loss/FBt2/assignment_loss‚0.06749443473431267 1loss/SRqqt2/detection_loss‚0.7294585594013333 2loss/SRqqt1/assignment_loss‚0.5153373306265032 %loss/total_loss‚8.723391586023704 % _timestamp‚1.7845102224822628e+09 _step‚312 /loss/FBt2/detection_loss‚0.9591722284796936 0loss/FRt1/assignment_loss‚1.7112832222897758 0loss/FRt2/assignment_loss‚0.7816256525510564 1loss/FBt1/assignment_loss‚0.03730960677238178 1loss/SRqqt1/detection_loss‚0.7373545944736507¸ G/_®‚(:Ô —îõÒâùÈË disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚20757.09228515625 network.recv‚885612( disk.nvme1c1n1.out‚10642.72802734375 disk.nvme1n1.in‚0! disk.dm-1.out‚5678.9208984375 disk.nvme0c0n1.in‚ 0.00390625' proc.memory.availableMB‚ 492.3984375# proc.memory.rssMB‚ 3011.16015625* proc.memory.percent‚18.378663063049316) disk./.usagePercent‚76.00754422757899 disk.nvme1c1n1.in‚0& disk.nvme0n1.out‚20757.09228515625$ memory_percent‚96.99463844299316 cpu‚25.629039062030426 proc.cpu.threads‚17 network.sent‚ 5.853551e+06% disk./.usageGB‚1166.9058227539062& disk.nvme1n1.out‚10642.72802734375 disk.nvme0n1.in‚ 0.00390625 disk.md0.in‚0 disk.md0.out‚ 111.62109375 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚25483.24609375 disk.dm-1.in‚0[z诂(:’ —îõÒú®ãÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.699' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.233000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]œï°‚(:Ò ¦îõÒÓÏÞË disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚20757.86279296875& disk.nvme1n1.out‚10643.27587890625 disk.dm-0.in‚ 0.00390625* proc.memory.percent‚18.378663063049316 cpu‚25.628400029039973 network.sent‚ 5.917041e+06) disk./.usagePercent‚76.00754546994904% disk./.usageGB‚1166.9058418273926! disk.dm-1.out‚5678.9833984375 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0c0n1æ*©ýn.in‚ 0.00390625( disk.nvme0c0n1.out‚20757.86279296875( disk.nvme1c1n1.out‚10643.27587890625 disk.nvme1n1.in‚0 disk.md0.out‚ 111.93359375$ memory_percent‚96.99463844299316# proc.memory.rssMB‚ 3011.16015625 network.recv‚889945 disk.dm-1.in‚0 disk.md0.in‚0 disk.dm-0.out‚ 25483.828125' proc.memory.availableMB‚ 492.3984375&©ž“±‚(:ˆ ¦îõÒÿ•¢Ì gpu.0.gpu‚56.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚289.365' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.455 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,'ß[²‚(:Ð µîõÒ¡ƒºË disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme0n1.out‚21013.0830078125 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚10772.54931640625& disk.nvme1n1.out‚10772.54931640625 network.sent‚ 5.921832e+06) disk./.usagePercent‚76.00677072798035 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚112.4375 disk.dm-0.out‚25867.09765625" disk.dm-1.out‚5679.12646484375# proc.memory.rssMB‚ 3011.16015625 disk.dm-1.in‚0( proc.memory.availableMB‚ 492.37109375 proc.cpu.threads‚17 disk.rbd0.out‚0' disk.nvme0c0n1.out‚21013.0830078125 disk.nvme0n1.in‚ 0.00390625 disk.dm-0.in‚ 0.00390625$ memory_percent‚96.99480533599854* proc.memory.percent‚18.378663063049316 cpu‚25.627701550639102 network.recv‚893305% disk./.usageGB‚1166.8939476013184J­Qb“³‚(:ˆ µîõÒéÊùË gpu.0.gpu‚91.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚275.205' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.735 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çsƒ2´- _wandbruntime‚3438  _runtime‚3438¢{ÁǵÁ _step‚313 & lr-AdamW/pg1‚0.0005687237812076731 & lr-AdamW/pg2‚0.0005687237812076731 trainer/global_step‚3849 % _timestamp‚1.7845102651163611e+09 _runtime‚3438.691965778v“ŒÍÌ¶Æ % _timestamp‚1.7845102651163611e+09 _runtime‚3438.691965778 _step‚313 & lr-AdamW/pg1‚0.0005687237812076731 & lr-AdamW/pg2‚0.0005687237812076731 trainer/global_step‚3849¹³P2·- _wandbruntime‚3438  _runtime‚3438¬º5Gä'¸ÞO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15931598 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69161119 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 trainer/global_step‚3849 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31112227 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4488232015360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚ 3438.69306003 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚592835924204544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚592836282668032 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4456022044188 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚592836282668032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132563728369664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38048894 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _step‚314 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9213310 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4455681482914 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚589390601419346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚584934919936432 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38059037 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚588373503182336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38059037 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9213041 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6718557 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6718560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4462762446336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4462421022208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31112225 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚588373520221696 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚592835924204544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69171264 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845102651174562e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4462421022208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132563608682496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4488130940416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚132563728369664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38048894 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _step‚314 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9213310 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4455681482914 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚589390601419346 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚584934919936432 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38059037 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚588373503182336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38059037 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9213041 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6718557 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6718560 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4462762446336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4462421022208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31112225 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚588373520221696 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚592835924204544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69171264 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845102651174562e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4462421022208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132563608682496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4488130940416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚ 3438.69306003 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚592835924204544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚592836282668032 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4456022044188 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚592836282668032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272ºáû6<2º- _wandbruntime‚3439  _runtime‚3439x(MùÇ%»éM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚592985667276288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4463872552448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4463531121152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9215299 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚589539498075816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15935590 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132598025382400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69188727 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6720294 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31120020 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15935877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128108658012160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69178568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚588521777684480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132597899410944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69188727 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9215583 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31120018 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚592985667276288 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31120018 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6720291 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38068707 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4489348758016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4456789871166 % _timestamp‚1.7845102660476654e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38058550 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69178568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38068707 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚585082367643264 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4463872552448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactU½ð¥(ive_split_bytes.all.current‚ 125971456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚588521777684480ÁzIô&¼îM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚588521777684480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132597899410944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69188727 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9215583 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31120018 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚592985667276288 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31120018 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6720291 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38068707 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4489348758016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4456789871166 % _timestamp‚1.7845102660476654e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38058550 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69178568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38068707 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚585082367643264 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4463872552448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚588521777684480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚592985667276288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4463872552448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4463531121152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9215299 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚589539498075816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15935590 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132598025382400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69188727 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6720294 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31120020 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15935877 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128108658012160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69178568»:\&2½- _wandbruntime‚3439  _runtime‚3439v•ì ¾‚ $ _timestamp‚1.784510266053052e+09 /loss/FBt1/detection_loss‚0.5060021669820084 %loss/total_loss‚8.650483518919222 2loss/SRqqt2/assignment_loss‚0.5884844906156548 2loss/FBt1/assignment_loss‚0.003911875326435269 1loss/FBt2/assignment_loss‚0.01630806705306152 1loss/SRqqt1/detection_loss‚0.7012798317503466 trainer/global_step‚3849 0loss/FRt1/detection_loss‚0.49081972433229115 epoch‚9 _step‚316 1loss/SRqqt2/detection_loss‚0.7478030929082801 0loss/FRt1/assignment_loss‚1.8345342070732302 0loss/FRt2/assignment_loss‚0.8692455096753733 2loss/SRqqt1/assignment_loss‚0.5327470577505683 /loss/FRt2/detection_loss‚1.6881222292551827 _runtime‚3439.628479279 /loss/FBt2/detection_loss‚0.6712252661967889›Y ¿‡ 1loss/SRqqt2/detection_loss‚0.7478030929082801 0loss/FRt1/assignment_loss‚1.8345342070732302 0loss/FRt2/assignment_loss‚0.8692455096753733 2loss/SRqqt1/assignment_loss‚0.5327470577505683 /loss/FRt2/detection_loss‚1.6881222292551827 _runtime‚3439.628479279 /loss/FBt2/detection_loss‚0.6712252661967889 $ _timestamp‚1.784510266053052e+09 /loss/FBt1/detection_loss‚0.5060021669820084 %loss/total_loss‚8.650483518919222 2loss/SRqqt2/assignment_loss‚0.5884844906156548 2loss/FBt1/assignment_loss‚0.003911875326435269 1loss/FBt2/assignment_loss‚0.01630806705306152 1loss/SRqqt1/detection_loss‚0.7012798317503466 trainer/global_step‚3849 0loss/FRt1/detection_loss‚0.49081972433229115 epoch‚9 _step‚316¼”/áz]À‚(:Ò ÄîõÒ²ÂÆË disk.rbd0.out‚0& disk.nvme0n1.out‚21401.00537109375 disk.nvme1c1n1.in‚0$ memory_percent‚96.57788276672363' proc.memory.availableMB‚ 560.6796875# proc.memory.rssMB‚ 2830.24609375 network.sent‚ 5.937634e+06 disk.md0.out‚ 112.98828125 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚10871.8779296875 disk.nvme1n1.in‚0) proc.memory.percent‚17.27445125579834) disk./.usagePercent‚76.00453023781627% disk./.usageGB‚1166.8595504760742 disk.dm-0.out‚26353.10546875 disk.dm-1.in‚0% disk.nvme1n1.out‚10871.8779296875 disk.md0.in‚0" disk.dm-1.out‚5679.20458984375( disk.nvme0c0n1.out‚21401.00537109375 cpu‚25.626931029436648 proc.cpu.threads‚17 network.recv‚896667 disk.rbd0.in‚0äÒeTžÁ‚(:“ ÄîõÒ’Ó‰Ì gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.212' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.07066666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+{ñ ^‚(:Ó ÓîõÒÔð½Ë disk.md0.in‚0! disk.dm-0.out‚26359.240234375 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2830.24609375) proc.memory.percent‚17.27445125579834 network.sent‚ 5.987073e+06% disk./.usageGB‚1166.8596954345703 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚21406.72216796875 disk.dm-1.in‚0! disk.dm-1.out‚5679.2685546875 cpu‚25.6269597182677 proc.cpu.threads‚17 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚21406.72216796875 disk.nvme1c1n1.in‚0 disk.md0.out‚ 113.36328125 network.recv‚900914( disk.nvme1c1n1.out‚10873.18798828125& disk.nvme1n1.out‚10873.18798828125 disk.dm-0.in‚ 0.00390625$ memory_percent‚96.57795429229736( proc.memory.availableMB‚ 560.66796875) disk./.usagePercent‚76.00453967982871 disk.nvme1n1.in‚01‚IžÃ‚(:“ ÓîõÒžÿ–Ì gpu.0.gpu‚27.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.776' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.25866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂýœXÄ‚(:Í âîõÒ¿¸¤Ë proc.cpu.threads‚17 network.sent‚ 5.993902e+06 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625! disk.dm-1.out‚5679.3310546875# disk.nvme0c0n1.out‚ 21408.640625 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚10873.67236328125 disk.nvme1n1.in‚0) disk./.usagePercent‚76.00456875128809% disk./.usageGB‚1166.8601417541504 disk.dm-1.in‚0$ memory_percent‚96.57800197601318) proc.memory.percent‚17.27445125579834 cpu‚25.627919622506838 disk.md0.out‚ 113.60546875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚10873.67236328125( proc.memory.availableMB‚ 560.66015625 network.recv‚904377# disk.dm-0.out‚26361.04150390625 disk.rbd0.out‚0! disk.nvme0n1.out‚ 21408.640625# proc.memory.rssMB‚ 2830.24609375§±àžÅ‚(:“ âîõÒÂæèË gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚272.233' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.74433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_5çž2Æ- _wandbruntime‚3481  _runtime‚3481"¢ÇÇÁ % _timestamp‚1.7845103083696334e+09 _runtime‚3481.945312279 _step‚317 & lr-AdamW/pg1‚0.0005674224403223714 & lr-AdamW/pg2‚0.0005674224403223714 trainer/global_step‚3899žÅh²ÌÈÆ & lr-AdamW/pg1‚0.0005674224403223714 & lr-AdamW/pg2‚0.0005674224403223714 trainer/global_step‚3899 % _timestamp‚1.7845103083696334e+09 _runtime‚3481.945312279 _step‚317½º™j%2É- _wandbruntime‚3481  _runtime‚3481H6€žã'ÊÝO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚595787228289536 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6805260 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚592306453310032 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70045566 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38543689 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4518331548160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚600305576877056 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚600305218413568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4511164696322 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚600305218413568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129734509858304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784510308370411e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3899 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚595787245328896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚161340268rk›_ËâO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚ 3481.94592324 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚600305218413568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4511164696322 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚600305218413568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚129734509858304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784510308370411e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3899 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚595787245328896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16134026 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚595787228289536 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6805260 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚592306453310032 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70045566 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38543689 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4518331548160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚600305576877056 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4545166882816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38553359 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16138337 _step‚319 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70052870 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31509668 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134312857011200 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70063029 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚600454961485312 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70052870 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9331343 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚595935519831040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚595935519831040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4519441654272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4545061620736 % _timestamp‚1.7845103089824274e+09 _runtime‚ 3482.55821776 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38543202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚3899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚595935502791680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚596966157062078 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4519100222976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16138065 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6806991 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70063029 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4512273084574 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚592453901016864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ýñ™dó&ÎíM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4545166882816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38553359 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16138337 _step‚319 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70052870 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31509668 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134312857011200 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70063029 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚600454961485312 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70052870 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9331343 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚595935519831040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚595935519831040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4519441654272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4545061620736 % _timestamp‚1.7845103089824274e+09 _runtime‚ 3482.55821776 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38543202 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚3899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚595935502791680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚596966157062078 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4519100222976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16138065 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6806991 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70063029 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4512273084574 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚592453901016864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚595935502791680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚592453883977504 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38553359 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4519441654272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296¿"kø†2Ï- _wandbruntime‚3482  _runtime‚3482Kºc Ѓ %loss/total_loss‚8.552912905234294 _runtime‚3482.563017788 /loss/FRt1/assignment_loss‚1.809671030849962 0loss/FRt2/assignment_loss‚0.8827874056205222 /loss/FBt2/detection_loss‚0.5465176411229988 trainer/global_step‚3899 % _timestamp‚1.7845103089839664e+09 2loss/SRqqt2/assignment_loss‚0.5800381443025702 3loss/FBt2/assignment_loss‚0.0016319196813269785 /loss/FRt2/detection_loss‚1.6113385590703189 1loss/SRqqt2/detection_loss‚0.6606785149398441 epoch‚9 /loss/FRt1/detection_loss‚0.5092752224583135 _step‚320 2loss/SRqqt1/assignment_loss‚0.5394190042933167 2loss/FBt1/assignment_loss‚0.009012398154206474 1loss/SRqqt1/detection_loss‚0.7304646944525036 /loss/FBt1/detection_loss‚0.6720783702884113)ÍHш /loss/FRt1/assignment_loss‚1.809671030849962 0loss/FRt2/assignment_loss‚0.8827874056205222 /loss/FBt2/detection_loss‚0.5465176411229988 trainer/global_step‚3899 % _timestamp‚1.7845103089839664e+09 2loss/SRqqt2/assignment_loss‚0.5800381443025702 3loss/FBt2/assignment_loss‚0.0016319196813269785 /loss/FRt2/detection_loss‚1.6113385590703189 1loss/SRqqt2/detection_loss‚0.6606785149398441 epoch‚9 /loss/FRt1/detection_loss‚0.5092752224583135 _step‚320 2loss/SRqqt1/assignment_loss‚0.5394190042933167 2loss/FBt1/assignment_loss‚0.009012398154206474 1loss/SRqqt1/detection_loss‚0.7304646944525036 /loss/FBt1/detection_loss‚0.6720783702884113 %loss/total_loss‚8.552912905234294 _runtime‚3482.563017788Àõ}[YÒ‚(:Î ñîõÒ·ÙõË disk.nvme0c0n1.in‚ 0.00390625& disk.nvme1n1.out‚10875.58349609375 disk.md0.out‚ 114.20703125 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 560.63671875 network.recv‚907807) disk./.usagePercent‚76.00456875128809% disk.nvme0n1.out‚21413.9873046875 disk.dm-0.out‚ 26366.875# proc.memory.rssMB‚ 2830.24609375% disk./.usageGB‚1166.8601417541504 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0" disk.dm-1.out‚5679.47900390625) proc.memory.percent‚17.27445125579834' disk.nvme0c0n1.out‚21413.9873046875 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚10875.58349609375 disk.md0.in‚0$ memory_percent‚96.57814502716064 cpu‚25.628431469638578 proc.cpu.threads‚17 network.sent‚ 6.009662e+06°2žÓ‚(:“ ñîõÒ‚’´Ì gpu.0.gpu‚55.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.811' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.27033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûé -XÔ‚(:Í €ïõÒ¥«°Ë network.recv‚911649) disk./.usagePercent‚76.00452278359592" disk.dm-0.out‚26368.1591796875# proc.memory.rssMB‚ 2830.24609375 cpu‚25.62952400023859 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚96.57816886901855 network.sent‚ 6.061026e+06% disk./.usageGB‚1166.8594360351562 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚5679.52197265625 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚21415.69189453125) proc.memory.percent‚17.27445125579834" disk.nvme1n1.out‚ 10876.3515625 disk.md0.in‚0 disk.md0.out‚ 114.74609375 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚21415.69189453125 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 10876.3515625' proc.memory.availableMB‚ 560.6328125, ¤˜žÕ‚(:“ €ïõÒùçýË gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚280.528' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.50933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä#‹p2Ö- _wandbruntime‚3524  _runtime‚3524³”?Æ×À & lr-AdamW/pg1‚0.0005661043887076932 & lr-AdamW/pg2‚0.0005661043887076932 trainer/global_step‚3949 $ _timestamp‚1.784510351237018e+09 _runtime‚3524.814045255 _step‚321~án]ËØÅ _runtime‚3524.814045255 _step‚321 & lr-AdamW/pg1‚0.0005661043887076932 & lr-AdamW/pg2‚0.0005661043887076932 trainer/global_step‚3949 $ _timestamp‚1.784510351237018e+09Á!Ë2Ù- _wandbruntime‚3524  _runtime‚3524˜äœëÉÚÝO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6891960 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6891957 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚603200953396736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607774512622592 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131393504750080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4599757662208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607774512622592 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131393504750080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4599757662208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚603349244938240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9445899 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31899318 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136027563839488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4600973595648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4575010756096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39038011 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16339323 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4600878819328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39038011 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4574669324800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4575010756096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚607924255694336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚604393531249832 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚599825417351104 trainer/global_step‚3949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚603349244938240 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6893691 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4568096859368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31899320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136027677228032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131426685020160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70937331 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚323 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39027854 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845103518486636e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70937331 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70927172 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6893694¯Âm¾àíM FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31899318 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136027563839488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4600973595648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4575010756096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39038011 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16339323 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4600878819328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39038011 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4574669324800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4575010756096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚607924255694336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚604393531249832 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚599825417351104 trainer/global_step‚3949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚603349244938240 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6893691 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4568096859368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31899320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136027677228032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131426685020160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70937331 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚323 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39027854 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845103518486636e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70937331 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70927172 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6893694 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚599825434390464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3525.424466907 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31899318 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚603349244938240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9445899 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ã)‹0 2á- _wandbruntime‚3525  _runtime‚3525à!þâ‚ 0loss/FRt1/assignment_loss‚1.6694735183169962 2loss/SRqqt2/assignment_loss‚0.5528121613473195 /loss/FRt2/detection_loss‚1.6604328077011319 /loss/FBt1/detection_loss‚0.5197653141838791 trainer/global_step‚3949 _step‚324 0loss/FRt2/assignment_loss‚0.9180163726210594 2loss/SRqqt1/assignment_loss‚0.5862429654568063 2loss/FBt2/assignment_loss‚0.022538554307118153 1loss/SRqqt1/detection_loss‚0.6151580011656571 % _timestamp‚1.7845103518499277e+09 1loss/SRqqt2/detection_loss‚0.7268645343789831 %loss/total_loss‚8.778986175726331 epoch‚9 _runtime‚3525.425179808 1loss/FBt1/assignment_loss‚0.00785036281575552 /loss/FRt1/detection_loss‚0.5084699781371097 /loss/FBt2/detection_loss‚0.9913616052945144†¿Ë ㇠/loss/FBt1/detection_loss‚0.5197653141838791 trainer/global_step‚3949 _step‚324 0loss/FRt2/assignment_loss‚0.9180163726210594 2loss/SRqqt1/assignment_loss‚0.5862429654568063 2loss/FBt2/assignment_loss‚0.022538554307118153 1loss/SRqqt1/detection_loss‚0.6151580011656571 % _timestamp‚1.7845103518499277e+09 1loss/SRqqt2/detection_loss‚0.7268645343789831 %loss/total_loss‚8.778986175726331 epoch‚9 _runtime‚3525.425179808 1loss/FBt1/assignment_loss‚0.00785036281575552 /loss/FRt1/detection_loss‚0.5084699781371097 /loss/FBt2/detection_loss‚0.9913616052945144 0loss/FRt1/assignment_loss‚1.6694735183169962 2loss/SRqqt2/assignment_loss‚0.5528121613473195 /loss/FRt2/detection_loss‚1.6604328077011319ÄB¾HåVä‚(:Ë žïõÒ˜©ËË% disk./.usageGB‚1166.8594779968262 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0( disk.nvme1c1n1.out‚10877.90673828125 proc.cpu.threads‚17 network.sent‚ 6.128507e+06 network.recv‚919162) disk./.usagePercent‚76.00452551681003& disk.nvme1n1.out‚10877.90673828125 disk.md0.out‚ 115.65625 disk.dm-0.out‚ 26370.6171875 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚21418.7568359375 disk.dm-1.in‚0" disk.dm-1.out‚5679.72802734375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚21418.7568359375$ memory_percent‚96.58145904541016% proc.memory.availableMB‚ 560.09375# proc.memory.rssMB‚ 2830.24609375 disk.nvme1c1n1.in‚0) proc.memory.percent‚17.27445125579834 cpu‚25.62807671719435V\wÑžå‚(:“ žïõÒú´Ì gpu.0.gpu‚94.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.297' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.43233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊeŽ6Wæ‚(:Ì ­ïõÒ€ÉË$ disk.nvme1n1.out‚10878.509765625 disk.md0.in‚0 disk.rbd0.in‚0& disk.nvme1c1n1.out‚10878.509765625$ memory_percent‚96.58527374267578% proc.memory.availableMB‚ 559.46875# proc.memory.rssMB‚ 2830.87109375 cpu‚25.62753899658416 disk.nvme1c1n1.in‚0 network.sent‚ 6.167552e+06 disk.dm-1.in‚0' disk.nvme0c0n1.out‚21420.5810546875 disk.nvme1n1.in‚0 disk.md0.out‚ 115.96875 disk.dm-0.in‚ 0.00390625* proc.memory.percent‚17.278265953063965) disk./.usagePercent‚76.00452800155017 disk.nvme0n1.in‚ 0.00390625# disk.dm-0.out‚26372.29248046875 proc.cpu.threads‚17% disk.nvme0n1.out‚21420.5810546875 network.recv‚923209% disk./.usageGB‚1166.8595161437988" disk.dm-1.out‚5679.79931640625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625¾†Ðžç‚(:“ ­ïõÒ¹”™Ì gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚283.621' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.54033333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;Úv2è- _wandbruntime‚3567  _runtime‚3567È·ÆýÇéÁ trainer/global_step‚3999 % _timestamp‚1.7845103936536632e+09 _runtime‚3567.229299561 _step‚325 & lr-AdamW/pg1‚0.0005647697110196306 & lr-AdamW/pg2‚0.0005647697110196306lG2ÓÌêÆ & lr-AdamW/pg1‚0.0005647697110196306 & lr-AdamW/pg2‚0.0005647697110196306 trainer/global_step‚3999 % _timestamp‚1.7845103936536632e+09 _runtime‚3567.229299561 _step‚325ÅOhC<2ë- _wandbruntime‚3567  _runtime‚3567hͪå'ìßO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6978657 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137708116501504 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845103936571426e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39512993 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32281175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6978660 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚610614695543296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚610614695543296 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9557272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32281177 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚610614678503936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71794170 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133052536866304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4655709808128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚615244165295104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚611671651180370 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚610614678503936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16536211 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚615243806831616 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4629128327680 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9557551 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137708246674432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚3999 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4629128327680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4629469751808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32281175 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4655598247424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39502850 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16535929 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚615243806831616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4629469751808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚615244165295104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39512993 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184ÚÔ¥ê'íäO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39502850 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16535929 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚615243806831616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4629469751808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚615244165295104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39512993 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6978657 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137708116501504 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845103936571426e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39512993 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32281175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6978660 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚610614695543296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚610614695543296 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9557272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32281177 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚610614678503936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71794170 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133052536866304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4655709808128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚615244165295104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚611671651180370 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚610614678503936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16536211 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚615243806831616 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4629128327680 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9557551 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137708246674432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚3999 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4629128327680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4629469751808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32281175 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4655598247424ÆÓñèã2î- _wandbruntime‚3567  _runtime‚3567A×/­3 ïéM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚610762953006080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71811633 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39512506 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16539925 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4630238426624 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6980394 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4656712276992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6980391 T‰&›÷¼FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 _runtime‚3567.844693797 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71811633 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4623239511390 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚611820547836840 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32288970 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _step‚327 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚610762970045440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137742410801152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7845103942688947e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71801474 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9559815 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133085717136384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4656821733376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚607196967764064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3999 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚611820190236094 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32288968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32288970 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137742538869760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4623580072776 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚615393549903360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4630238426624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32288968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133085698524160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39522663 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚610762953006080 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39522663 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚607196950724704 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16540209 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9559534 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4630579857920xqDWô&ðîM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚615393549903360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4630238426624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32288968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133085698524160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39522663 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚610762953006080 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39522663 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚607196950724704 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16540209 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9559534 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4630579857920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚610762953006080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71811633 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39512506 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16539925 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4630238426624 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6980394 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4656712276992 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6980391 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 _runtime‚3567.844693797 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71811633 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4623239511390 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚611820547836840 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32288970 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _step‚327 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚610762970045440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137742410801152 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7845103942688947e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71801474 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9559815 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133085717136384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4656821733376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚607196967764064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3999 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚611820190236094 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32288968 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32288970 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137742538869760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4623580072776 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÇKÚÀ2ñ- _wandbruntime‚3567  _runtime‚3567è?(òþ 2loss/FBt2/assignment_loss‚0.007812321266960546 1loss/SRqqt1/detection_loss‚0.7900761054346872 trainer/global_step‚3999 _step‚328 0loss/FRt1/assignment_loss‚1.7020949345440668 .loss/FRt1/detection_loss‚0.531772828737042 1loss/SRqqt2/detection_loss‚0.7308110374874195 2loss/SRqqt1/assignment_loss‚0.5171580502400412 1loss/SRqqt2/assignment_loss‚0.618496034842078 /loss/FBt1/detection_loss‚0.6364376927948022 /loss/FBt2/detection_loss‚0.7457114477052956 $loss/total_loss‚8.58930038430919 epoch‚9 $ _timestamp‚1.784510394271004e+09 0loss/FRt2/assignment_loss‚0.8492314779648075 2loss/FBt1/assignment_loss‚0.008257554926752262 /loss/FRt2/detection_loss‚1.4514408983652376 _runtime‚ 3567.84671405Dƒü óƒ 2loss/FBt2/assignment_loss‚0.007812321266960546 1loss/SRqqt1/detection_loss‚0.7900761054346872 trainer/global_step‚3999 _step‚328 0loss/FRt1/assignment_loss‚1.7020949345440668 .loss/FRt1/detection_loss‚0.531772828737042 1loss/SRqqt2/detection_loss‚0.7308110374874195 2loss/SRqqt1/assignment_loss‚0.5171580502400412 1loss/SRqqt2/assignment_loss‚0.618496034842078 /loss/FBt1/detection_loss‚0.6364376927948022 /loss/FBt2/detection_loss‚0.7457114477052956 $loss/total_loss‚8.58930038430919 epoch‚9 $ _timestamp‚1.784510394271004e+09 0loss/FRt2/assignment_loss‚0.8492314779648075 2loss/FBt1/assignment_loss‚0.008257554926752262 /loss/FRt2/detection_loss‚1.4514408983652376 _runtime‚ 3567.84671405ȃu5]ô‚(:Ò ¼ïõÒÕØåË& disk.nvme0n1.out‚21421.96142578125! disk.dm-0.out‚26373.416015625 disk.nvme0n1.in‚ 0.00390625# memory_percent‚97.0787763595581( proc.memory.availableMB‚ 478.61328125# proc.memory.rssMB‚ 2911.72265625) proc.memory.percent‚17.77174472808838 proc.cpu.threads‚17 network.sent‚ 6.186306e+06 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21421.96142578125 cpu‚25.629720940612977 network.recv‚926381% disk./.usageGB‚1166.8591041564941 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 116.3203125" disk.dm-1.out‚5679.83056640625 disk.rbd0.in‚0) disk./.usagePercent‚76.00450116635689' disk.nvme1c1n1.out‚10879.0556640625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚10879.0556640625 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625i<Òœõ‚(:‘ ¼ïõÒé ¦Ì gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.765' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.92166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´X[ö‚(:Ð ËïõÒÕ¡ÛË disk.rbd0.in‚0& disk.nvme0n1.out‚21425.89697265625( proc.memory.availableMB‚ 447.77734375 disk.dm-1.out‚5679.978515625 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚21425.89697265625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0& disk.nvme1n1.out‚10880.07080078125 network.sent‚ 6.236978e+06 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.md0.out‚ 116.8046875$ memory_percent‚97.26698398590088 cpu‚25.642302375171724 proc.cpu.threads‚17% disk./.usageGB‚1164.1419105529785 disk.rbd0.out‚0( disk.nvme1c1n1.out‚10880.07080078125 disk.md0.in‚0" proc.memory.rssMB‚ 2942.0234375) proc.memory.percent‚17.95668601989746 network.recv‚931209) disk./.usagePercent‚75.82751412167259 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚ 26377.1796875ZÕgœ÷‚(:‘ ËïõÒ³ÑšÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚219.002' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚73.00066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øõé‡2ø- _wandbruntime‚3585  _runtime‚3585‰ûç¢+ù„{ 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6444668173789978 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6789116859436035 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 particle/accuracy_1_of_5‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 +jet/accuracy_4_of_4‚0.01670061983168125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7543104887008667 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6866668462753296 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1954318881034851 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.65422523021698 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 6validation_average_jet_accuracy‚0.3602391481399536 _runtime‚3585.469683313 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3833308815956116 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7342429161071777 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02810371294617653 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15726357698440552 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1478753387928009 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879648089408875 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9952901005744934 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925970077515 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03255429491400719 *jet/accuracy_1_of_2‚0.7851227521896362 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21333187818527222 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982199668884277 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.578883171081543 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04104618728160858 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12502939999103546 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20076629519462585 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_3_of_3‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6096576452255249 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_4_of_5‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19249825179576874 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2974638044834137 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04399040341377258 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7934324145317078 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6851817965507507 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0395379476249218 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7379052042961121 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.849092960357666 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16429032385349274 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7238633036613464 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9637017250061035 +jet/accuracy_3_of_4‚0.09828110039234161 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.1538137048482895 'particle/f_score‚0.6508188247680664 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13547714054584503 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7621315717697144 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8407135009765625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30001989006996155 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13426940143108368 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1563969999551773 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 (particle/accuracy‚0.8510936498641968 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6488566398620605 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.591935932636261 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5036656260490417 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4779984951019287 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5787695646286011 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.649967610836029 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7557598948478699 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6761711239814758 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18936380743980408 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 jet/accuracy_3_of_5‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5799999833106995 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7572944760322571 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960847496986389 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6962108016014099 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.209333598613739 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877771735191345 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8425925970077515 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987993836402893 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6901041269302368 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.07391560077667236 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6935448050498962 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8424414396286011 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 *jet/accuracy_2_of_3‚0.5596132278442383 *jet/accuracy_2_of_4‚0.8028932213783264 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8364512324333191 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1083659902215004 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.742675244808197 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020409563556313515 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22371630370616913 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_1_of_3‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6507615447044373 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_2_of_3‚1 _step‚329 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941513538360596 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17835837602615356 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04491632804274559 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009831513278186321 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 trainer/global_step‚3999 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8719775080680847 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1842673122882843 +particle/sensitivity‚0.6462397575378418 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37985390424728394 (loss/val_total_loss‚8.08969808815199 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1804819405078888 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6310103535652161 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17499999701976776 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8407135009765625 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7348753809928894 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7493295073509216 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17208987474441528 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8847227692604065 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt0SRqqt2FBt/eve„3c±ènt_purity‚0.9824140667915344 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03255429491400719 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7296670079231262 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982174634933472 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7932142615318298 +jet/accuracy_1_of_1‚0.31325095891952515 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4842517375946045 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17835837602615356 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04142332077026367 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16687284409999847 epoch‚9 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6468064188957214 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6609463691711426 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9205285906791687 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_5_of_5‚0 $ _timestamp‚1.784510411893651e+09 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018161313608288765 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2606595754623413 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4729439914226532 5 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.85309898853302 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46480002999305725 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966185688972473 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950701594352722 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982174634933472 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6718707084655762 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9729623198509216 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4270833432674408 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9467478394508362 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6766576170921326 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959597587585449 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8523242473602295 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03868873789906502 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.63402259349823 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7177640795707703 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46438997983932495 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7396063208580017 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 jet/accuracy_5_of_5‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.171458899974823 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9062937498092651 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7563961744308472 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11660066992044449 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6511427760124207 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17052412033081055 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19224053621292114 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24702146649360657 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.662900447845459 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6488566398620605 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +particle/specificity‚0.9071170091629028]uB:=ú‰{ & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9637017250061035 +jet/accuracy_3_of_4‚0.09828110039234161 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.1538137048482895 'particle/f_score‚0.6508188247680664 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13547714054584503 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7621315717697144 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8407135009765625 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30001989006996155 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13426940143108368 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1563969999551773 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 (particle/accuracy‚0.8510936498641968 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6488566398620605 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.591935932636261 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5036656260490417 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4779984951019287 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5787695646286011 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.649967610836029 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7557598948478699 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6761711239814758 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18936380743980408 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 jet/accuracy_3_of_5‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5799999833106995 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7572944760322571 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960847496986389 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6962108016014099 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.209333598613739 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9877771735191345 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8425925970077515 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987993836402893 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6901041269302368 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.07391560077667236 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6935448050498962 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8424414396286011 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 *jet/accuracy_2_of_3‚0.5596132278442383 *jet/accuracy_2_of_4‚0.8028932213783264 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8364512324333191 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1083659902215004 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.742675244808197 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.020409563556313515 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22371630370616913 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_1_of_3‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6507615447044373 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_2_of_3‚1 _step‚329 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941513538360596 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17835837602615356 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04491632804274559 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009831513278186321 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 trainer/global_step‚3999 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8719775080680847 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1842673122882843 +particle/sensitivity‚0.6462397575378418 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37985390424728394 (loss/val_total_loss‚8.08969808815199 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1804819405078888 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6310103535652161 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17499999701976776 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8407135009765625 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7348753809928894 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7493295073509216 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17208987474441528 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8847227692604065 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03255429491400719 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7296670079231262 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982174634933472 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7932142615318298 +jet/accuracy_1_of_1‚0.31325095891952515 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4842517375946045 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17835837602615356 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04142332077026367 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16687284409999847 epoch‚9 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6468064188957214 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6609463691711426 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9205285906791687 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_5_of_5‚0 $ _timestamp‚1.784510411893651e+09 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018161313608288765 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2606595754623413 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4729439914226532 5 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.85309898853302 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46480002999305725 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966185688972473 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950701594352722 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982174634933472 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6718707084655762 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9729623198509216 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4270833432674408 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987978339195251 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9467478394508362 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6766576170921326 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959597587585449 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8523242473602295 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03868873789906502 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.63402259349823 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7177640795707703 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46438997983932495 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7396063208580017 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 jet/accuracy_5_of_5‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.171458899974823 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9062937498092651 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7563961744308472 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11660066992044449 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6511427760124207 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17052412033081055 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19224053621292114 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24702146649360657 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.662900447845459 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6488566398620605 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +particle/specificity‚0.9071170091629028 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6444668173789978 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6789116859436035 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 particle/accuracy_1_of_5‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 +jet/accuracy_4_of_4‚0.01670061983168125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7543104887008667 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6866668462753296 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1954318881034851 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.65422523021698 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 6validation_average_jet_accuracy‚0.3602391481399536 _runtime‚3585.469683313 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3833308815956116 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7342429161071777 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02810371294617653 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15726357698440552 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1478753387928009 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9879648089408875 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9952901005744934 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925970077515 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03255429491400719 *jet/accuracy_1_of_2‚0.7851227521896362 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21333187818527222 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982199668884277 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.578883171081543 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04104618728160858 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12502939999103546 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20076629519462585 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_3_of_3‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6096576452255249 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_4_of_5‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19249825179576874 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2974638044834137 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04399040341377258 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7934324145317078 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6851817965507507 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0395379476249218 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7379052042961121 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.849092960357666 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16429032385349274 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7238633036613464 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645É«b]IYû‚(:Î ÚïõÒÃÈçË disk.dm-0.out‚ 26377.6875 disk.dm-1.out‚5925.365234375( disk.nvme0c0n1.out‚21426.53076171875 disk.nvme1c1n1.in‚0# memory_percent‚95.4888105392456( disk./.usagePercent‚75.8275245575811% disk./.usageGB‚1164.1420707702637 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625( proc.memory.availableMB‚ 739.11328125 network.recv‚941038& disk.nvme0n1.out‚21426.53076171875# proc.memory.rssMB‚ 2945.16015625 cpu‚25.643715616733914 proc.cpu.threads‚17 network.sent‚ 6.256133e+06 disk.md0.out‚ 117.04296875 disk.dm-0.in‚ 0.00390625( disk.nvme1c1n1.out‚11125.85693359375* proc.memory.percent‚17.975831031799316 disk.nvme1n1.in‚0& disk.nvme1n1.out‚11125.85693359375 disk.md0.in‚0â¨eÇü‚(:’ ÚïõÒ¶ôµÌ gpu.0.gpu‚60.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚284.36' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.78666666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QøkøZý‚(:Ï éïõÒé§ÆË( disk.nvme0c0n1.out‚21431.53271484375 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚21431.53271484375 disk.dm-1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2945.16015625 cpu‚25.64532883585433( disk./.usagePercent‚75.8275523866704% disk./.usageGB‚1164.1424980163574& disk.nvme1n1.out‚11373.15478515625$ memory_percent‚95.48892974853516% proc.memory.availableMB‚ 739.09375* proc.memory.percent‚17.975831031799316 proc.cpu.threads‚17( disk.nvme1c1n1.out‚11373.15478515625 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.out‚26383.19921875! disk.dm-1.out‚6170.8369140625 disk.nvme0c0n1.in‚ 0.00390625 network.sent‚ 6.302078e+06 network.recv‚945628 disk.nvme1c1n1.in‚0 disk.md0.out‚ 117.6640625 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0V”¯£“þ‚(:ˆ éïõÒ¦Ì gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚120.426' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.142 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mð|úVÿ‚(:Ë øïõÒ€ø¼Ë network.recv‚949324' disk./.usagePercent‚76.007418251255( disk.nvme1c1n1.out‚11373.80322265625 disk.md0.in‚0# disk.dm-0.out‚28297.61181640625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625! disk.nvme0n1.out‚ 23346.328125 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 118.12890625 disk.dm-1.in‚0# disk.nvme0c0n1.out‚ 23346.328125( proc.memory.availableMB‚ 738.83203125% disk./.usageGB‚1166.9038887023926$ memory_percent‚95.49052715301514# proc.memory.rssMB‚ 2945.16015625* proc.memory.percent‚17.975831031799316 cpu‚25.64699469493098 proc.cpu.threads‚17 network.sent‚ 6.305423e+06& disk.nvme1n1.out‚11373.80322265625 disk.dm-0.in‚ 0.00390625! disk.dm-1.out‚6170.8837890625 disk.rbd0.out‚0ÅPÓ“€‚(:ˆ øïõÒ¢ß»Ë gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚288.144' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôç:ð2- _wandbruntime‚3631  _runtime‚3631áæ‰Æ‚À % _timestamp‚1.7845104575190194e+09 _runtime‚ 3631.09461007 _step‚330 & lr-AdamW/pg1‚0.0005634184929820375 & lr-AdamW/pg2‚0.0005634184929820375 trainer/global_step‚4049õUk˃Š& lr-AdamW/pg1‚0.0005634184929820375 & lr-AdamW/pg2‚0.0005634184929820375 trainer/global_step‚4049 % _timestamp‚1.7845104575190194e+09 _runtime‚ 3631.09461007 _step‚330ÊH~‘/2„- _wandbruntime‚3631  _runtime‚3631ç>ç^m…ßO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72872779 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4732424274432 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40069001 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16781312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _runtime‚3631.095752562 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9711836 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚621040227624504 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32793633 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139466553307136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624675762754560 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 % _timestamp‚1.7845104575200832e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7069476 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624676121218048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚621040585225138 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7069473 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4705015267840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9711571 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚616342326478416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40079144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚619970764526080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624675762754560 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134734110420480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139466429425664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segme”W~…xnt.small_pool.current‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚619970747486720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4705015267840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚619970747486720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40079144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72862634 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32793635 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚624676121218048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134734129032704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4732319005184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32793633 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4698258746722 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚331 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4705356691968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16781044GUtê'†äO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚331 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4705356691968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16781044 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72872779 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4732424274432 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40069001 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16781312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _runtime‚3631.095752562 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9711836 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚621040227624504 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32793633 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139466553307136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624675762754560 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 % _timestamp‚1.7845104575200832e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7069476 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624676121218048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚621040585225138 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7069473 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4705015267840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9711571 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚616342326478416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40079144 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚619970764526080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚624675762754560 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134734110420480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139466429425664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚619970747486720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4705015267840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚619970747486720 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40079144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72862634 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32793635 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚624676121218048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134734129032704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4732319005184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32793633 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4698258746722Ë< e2‡- _wandbruntime‚3631  _runtime‚3631î&ˆèM ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚624825505826304 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚620119039028224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9714150 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624825147355648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9713881 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚139500842701312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚620119021988864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4733442816512 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚616489757145888 trainer/global_step‚4049 % _timestamp‚1.7845104581501114e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16785360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32801428 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4706466798080 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4699026573700 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚621188766680228 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4699367135086 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72880083 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4706466798080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4733533398528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134767290690560 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚620119039028224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40088814 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72890242 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32801426 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40078657 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72890242 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚621189124280974 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32801426 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16785088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139500733507072 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624825505826304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32801428 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134767290690560 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚620119039028224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40088814 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72890242 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32801426 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40078657 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72890242 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚621189124280974 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32801426 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16785088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139500733507072 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624825505826304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32801428 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9713881 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚139500842701312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚620119021988864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4733442816512 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚616489757145888 trainer/global_step‚4049 % _timestamp‚1.7845104581501114e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16785360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32801428 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4706466798080 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303̾ýD2Š- _wandbruntime‚3631  _runtime‚3631.ÑàŸ(‹‚ 2loss/FBt2/assignment_loss‚0.005312242047427418 /loss/FRt1/detection_loss‚0.4919269747629616 1loss/SRqqt1/detection_loss‚0.6970252271776995 trainer/global_step‚4049 _step‚333 0loss/FRt1/assignment_loss‚1.7660218422697689 2loss/SRqqt1/assignment_loss‚0.4971962ÿð­îà826840043 2loss/FBt1/assignment_loss‚0.013928665500483403 1loss/SRqqt2/detection_loss‚0.6880419198608411 -loss/FBt2/detection_loss‚0.59605517295559 %loss/total_loss‚8.090254886313186 epoch‚10 0loss/FRt2/assignment_loss‚0.7684453565116022 2loss/SRqqt2/assignment_loss‚0.5023360024451644 /loss/FBt1/detection_loss‚0.5341469049046281 % _timestamp‚1.7845104581512132e+09 /loss/FRt2/detection_loss‚1.5298182951930144 _runtime‚3631.726513033qéí Œ‡ 2loss/FBt2/assignment_loss‚0.005312242047427418 /loss/FRt1/detection_loss‚0.4919269747629616 1loss/SRqqt1/detection_loss‚0.6970252271776995 trainer/global_step‚4049 _step‚333 0loss/FRt1/assignment_loss‚1.7660218422697689 2loss/SRqqt1/assignment_loss‚0.4971962826840043 2loss/FBt1/assignment_loss‚0.013928665500483403 1loss/SRqqt2/detection_loss‚0.6880419198608411 -loss/FBt2/detection_loss‚0.59605517295559 %loss/total_loss‚8.090254886313186 epoch‚10 0loss/FRt2/assignment_loss‚0.7684453565116022 2loss/SRqqt2/assignment_loss‚0.5023360024451644 /loss/FBt1/detection_loss‚0.5341469049046281 % _timestamp‚1.7845104581512132e+09 /loss/FRt2/detection_loss‚1.5298182951930144 _runtime‚3631.726513033ÍèKR6ž‚(:“ ‡ðõÒ蟯Ì gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.864' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.62133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µsÈgWŽ‚(:Ì ‡ðõÒÃÑÜÍ* proc.memory.percent‚17.975831031799316 cpu‚25.648613177066117 disk.rbd0.out‚0 network.sent‚ 6.327199e+06 disk.nvme1n1.in‚0 disk.md0.out‚ 118.4140625 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚28300.8291015625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 network.recv‚958941% disk./.usageGB‚1166.9044723510742! disk.dm-1.out‚6171.0595703125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 proc.cpu.threads‚17 disk.dm-1.in‚0) disk./.usagePercent‚76.00745626777882$ disk.nvme1n1.out‚11374.607421875 disk.md0.in‚0& disk.nvme1c1n1.out‚11374.607421875$ memory_percent‚95.49360275268555& proc.memory.availableMB‚ 738.328125& disk.nvme0c0n1.out‚23349.533203125$ disk.nvme0n1.out‚23349.533203125# proc.memory.rssMB‚ 2945.16015625*ã^‚(:Ó –ðõÒúåËË* proc.memory.percent‚17.975831031799316 network.recv‚964146 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚29168.0087890625( proc.memory.availableMB‚ 738.56640625% disk./.usageGB‚1166.8583526611328& disk.nvme1n1.out‚11501.48388671875 disk.md0.out‚ 118.71875 disk.dm-1.in‚0! disk.dm-1.out‚6171.1259765625 disk.rbd0.out‚0& disk.nvme0n1.out‚24090.56982421875( disk.nvme1c1n1.out‚11501.48388671875 network.sent‚ 6.376826e+06 disk.nvme1n1.in‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚24090.56982421875 disk.nvme1c1n1.in‚0$ memory_percent‚95.49214839935303 cpu‚25.64994172764392 proc.cpu.threads‚17) disk./.usagePercent‚76.00445221697657 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625# proc.memory.rssMB‚ 2945.16015625~=ÔT‚(:’ –ðõÒóÀˆÌ gpu.0.gpu‚19.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚181.62' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚60.540000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯8’Y‘‚(:Î ¥ðõÒš„ÒË proc.cpu.threads‚17 network.recv‚967632$ disk.nvme0n1.out‚24098.119140625 disk.md0.in‚0" disk.dm-1.out‚6171.27978515625 disk.rbd0.out‚0$ memory_percent‚95.49376964569092( proc.memory.availableMB‚ 738.30078125) disk./.usagePercent‚75.98896632266732 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0c0n1.out‚24098.119140625! disk.dm-0.out‚29175.689453125 disk.dm-1.in‚0# proc.memory.rssMB‚ 2945.16015625* proc.memory.percent‚17.975831031799316 network.sent‚ 6.37979e+06 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚11502.99853515625& disk.nvme1n1.out‚11502.99853515625 disk.md0.out‚ 119.296875 disk.rbd0.in‚0# disk./.usageGB‚1166.62060546875 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625 cpu‚25.650686251643446Sg$“Ÿ’‚(:” ¥ðõÒÞô¤Ì gpu.0.gpu‚50.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.666' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.221999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01Q’2“- _wandbruntime‚3675  _runtime‚3675§‘NÇ”Á & lr-AdamW/pg1‚0.0005620508213811234 & lr-AdamW/pg2‚0.0005620508213811234 trainer/global_step‚4099 % _timestamp‚1.7845105015928235e+09 _runtime‚3675.168469673 _step‚334/¦%jÌ•Æ & lr-AdamW/pg1‚0.0005620508213811234 & lr-AdamW/pg2‚0.0005620508213811234 trainer/global_step‚4099 % _timestamp‚1.7845105015928235e+09 _runtime‚3675.168469673 _step‚334έúM2–- _wandbruntime‚3675  _runtime‚3675Ñ>ä'—ÞO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚632145056963584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4753401398856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40553653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40563796 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4788199100928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 % _timestamp‚1.7845105015971732e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚623713842812656 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627384472593920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚632145056963584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33183285 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4760925793792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33183283 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚632145415427072 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚632145056963584 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4753401398856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40553653 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40563796 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4788199100928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 % _timestamp‚1.7845105015971732e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚623713842812656 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627384472593920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚632145056963584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33183285 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4760925793792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33183283 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚632145415427072 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4099 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33191076 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627532747096064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73764544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4789416369664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73754385 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40563309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4789311107584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4762035899904 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627532764135424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚623861273480128 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 % _timestamp‚1.7845105022081335e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚632294441564672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4754850348494 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40563309 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4754509787108 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33191078 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16986882 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7157910 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40573466 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16987175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40573466 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚627532764135424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136426304194560 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚632294800035328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73764544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚623861290519488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73754385 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141215739176448Ì^Þñ&›ëM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4762035899904 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627532764135424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚623861273480128 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 % _timestamp‚1.7845105022081335e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚632294441564672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4754850348494 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40563309 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4754509787108 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33191078 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16986882 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7157910 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40573466 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16987175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40573466 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚627532764135424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136426304194560 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚632294800035328 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73764544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚623861290519488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73754385 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141215739176448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9828975 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4099 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33191076 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627532747096064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73764544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4789416369664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73754385 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40563309 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4789311107584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ÐÔ!Ü)2œ- _wandbruntime‚3675  _runtime‚3675ÚÚyk€ epoch‚10 0loss/FRt2/assignment_loss‚0.8215335143133274 /loss/FRt1/detection_loss‚0.5281874875370615 .loss/FRt2/detection_loss‚1.632413258179299 /loss/SRqqt1/detection_loss‚0.78885371815128 1loss/SRqqt2/detection_loss‚0.6607414027698196 0loss/FRt1/assignment_loss‚1.7428190402567443 2loss/SRqqt2/assignment_loss‚0.5796346650948409 2loss/FBt2/assignment_loss‚0.003237149351567175 /loss/FBt2/detection_loss‚0.6691263332142539 $loss/total_loss‚8.45010914091669 trainer/global_step‚4099 2loss/SRqqt1/assignment_loss‚0.5649304348092189 1loss/FBt1/assignment_loss‚0.00316189298104289 0loss/FBt1/detection_loss‚0.45547024425823335 % _timestamp‚1.7845105022108316e+09 _runtime‚3675.786309314 _step‚337‹-· ž… /loss/FBt2/detection_loss‚0.6691263332142539 $loss/total_loss‚8.45010914091669 trainer/global_step‚4099 2loss/SRqqt1/assignment_loss‚0.5649304348092189 1loss/FBt1/assignment_loss‚0.00316189298104289 0loss/FBt1/detection_loss‚0.45547024425823335 % _timestamp‚1.7845105022108316e+09 _runtime‚3675.786309314 _step‚337 epoch‚10 0loss/FRt2/assignment_loss‚0.8215335143133274 /loss/FRt1/detection_loss‚0.5281874875370615 .loss/FRt2/detection_loss‚1.632413258179299 /loss/SRqqt1/detection_loss‚0.78885371815128 1loss/SRqqt2/detection_loss‚0.6607414027698196 0loss/FRt1/assignment_loss‚1.7428190402567443 2loss/SRqqt2/assignment_loss‚0.5796346650948409 2loss/FBt2/assignment_loss‚0.003237149351567175Ñ8èUT^Ÿ‚(:Ó ´ðõÒÓ÷âË" disk.dm-1.out‚6174.52978515625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚24099.43994140625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚11506.87744140625 disk.nvme1n1.in‚0& disk.nvme1n1.out‚11506.87744140625 network.recv‚970910 disk.md0.out‚ 119.796875# disk.dm-0.out‚29176.57666015625 disk.dm-1.in‚0& disk.nvme0n1.out‚24099.43994140625$ memory_percent‚95.49264907836914& proc.memory.availableMB‚ 738.484375# proc.memory.rssMB‚ 2945.16015625 network.sent‚ 6.397541e+06 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625* proc.memory.percent‚17.975831031799316 cpu‚25.651625912162615( disk./.usagePercent‚75.9889849582182$ disk./.usageGB‚1166.620891571045 disk.md0.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625_R¤ ‚(:’ ´ðõÒ€õžÌ gpu.0.gpu‚32.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚265.576' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.52533333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä7`¡‚(:Õ ÃðõÒ’î­Ë disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 120.19921875( proc.memory.availableMB‚ 738.46484375 proc.cpu.threads‚17! disk.dm-1.out‚6174.6826171875( disk.nvme0c0n1.out‚24211.53466796875 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚11540.94775390625 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚29321.71484375$ memory_percent‚95.49276828765869) disk./.usagePercent‚76.00447706437775 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚11540.94775390625* proc.memory.percent‚17.975831031799316 cpu‚25.651759749579476 network.sent‚ 6.447082e+06 network.recv‚976268% disk./.usageGB‚1166.8587341308594& disk.nvme0n1.out‚24211.53466796875# proc.memory.rssMB‚ 2945.16015625ú‡oAž¢‚(:“ ÃðõÒ¥õ‰Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚140.741' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚46.91366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P9ž2£- _wandbruntime‚3718  _runtime‚3718Ô^>%ǤÁ _runtime‚3718.197035227 _step‚338 & lr-AdamW/pg1‚0.0005606667840598797 & lr-AdamW/pg2‚0.0005606667840598797 trainer/global_step‚4149 % _timestamp‚1.7845105446204562e+09¢Ï¡Ì¥Æ trainer/global_step‚4149 % _timestamp‚1.7845105446204562e+09 _runtime‚3718.197035227 _step‚338 & lr-AdamW/pg1‚0.0005606667840598797 & lr-AdamW/pg2‚0.0005606667840598797ÒÌ &A2¦- _wandbruntime‚3718  _runtime‚3718)´´1å'§ßO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4808884612264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4816494895616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41048448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845105446211727e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚631085393225616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4844119238656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41038305 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74611238 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4816153471488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41038305 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚639614709636096 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7242873 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74621383 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 trainer/global_step‚4149 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚339 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9942378 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138052156040704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33572935 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚634798197701120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚639614709636096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚639614351172608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33572933 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚142896153495040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4816153471488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17184976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _runtime‚3718.197098097 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚634798214740480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚142896275279360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4844016066560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4816494895616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138052137428480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639614351172608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17185254 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33572935 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635894260798520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚635894618399154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184ó©zíg¨äO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4816153471488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17184976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _runtime‚3718.197098097 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚634798214740480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚142896275279360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4844016066560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4816494895616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138052137428480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639614351172608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17185254 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33572935 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635894260798520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚635894618399154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4808884612264 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4816494895616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41048448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845105446211727e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚631085393225616 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4844119238656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41038305 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74611238 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/alloca{L 8ƒted_bytes.small_pool.freed‚ 4816153471488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41038305 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚639614709636096 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7242873 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74621383 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 trainer/global_step‚4149 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚339 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9942378 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚138052156040704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33572935 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚634798197701120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚639614709636096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚639614351172608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33572933 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚142896153495040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128Ó) «ú2©- _wandbruntime‚3718  _runtime‚37187$n½ï&ªéM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚634946489242624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4817263570432 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7244610 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74638846 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚639764094244352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7244607 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚142930442978304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 % _timestamp‚1.7845105452449532e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33580728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4817605001728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚142930568949760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4809993000516 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17188984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚634946472203264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17189293 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚4149 _runtime‚3718.820676031 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4817605001728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33580728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33580726 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74628687 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚634946472203264 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _step‚340 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639764094244352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41047961 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41058118 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9944683 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4845232638976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639763735773696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚634946489242624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41047961 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚631232806853728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4810333561902 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360âþ¹Oô&«îM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74638846 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9944377 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚631232823893088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _step‚340 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639764094244352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41047961 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41058118 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9944683 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4845232638976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639763735773696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚634946489242624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41047961 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚631232806853728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4810333561902 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚634946489242624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4817263570432 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7244610 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74638846 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚639764094244352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7244607 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚142930442978304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 % _timestamp‚1.7845105452449532e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33580728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4817605001728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚142930568949760 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4809993000516 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17188984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚634946472203264 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17189293 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚4149 _runtime‚3718.820676031 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4817605001728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33580728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33580726 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74628687 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚634946472203264 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4899934615040 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17386578 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7329573 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚643321277385528 % _timestamp‚1.7845105883011231e+09 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚643321634986162 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚647083645381632 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚144611104159744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647083645381632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33962583 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚642211922808320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚647084003845120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚638456926599216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33962583 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75485540 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚642211939847680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17386862 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41522957 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41533100 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚642211939847680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41533100 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚4199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33962585 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7329576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647084003845120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3761.876717481 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0,òCÆë'ºåO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41522957 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41533100 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚642211939847680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41533100 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚4199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33962585 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7329576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647084003845120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3761.876717481 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4871722573312 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚144610986569728 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4899934615040 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17386578 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7329573 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚643321277385528 % _timestamp‚1.7845105883011231e+09 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚643321634986162 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚647083645381632 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚144611104159744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647083645381632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33962583 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚642211922808320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚647084003845120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚638456926599216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33962583 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75485540 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚642211939847680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17386862 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232׌ѳ62»- _wandbruntime‚3762  _runtime‚3762¸ºî&¼èM FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4901053841920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139744349814784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4865476213924 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647233388453376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33970378 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784510588926429e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7331307 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚638604340227328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚642360214349824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17390627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33970378 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚642360197310464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33970376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4900956968448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75513148 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75502989 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4872832672256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647233029982720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17390924 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4873174103552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10059320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144645403656704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41542770 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41532613 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7331310 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚638604357266688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10059614 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75513148 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚643469816441252 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144645288171008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528˜í^Ç ½íM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75513148 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75502989 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4872832672256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647233029982720 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17390924 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4873174103552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10059320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144645403656704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41542770 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41532613 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7331310 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚638604357266688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10059614 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75513148 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚643469816441252 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144645288171008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 2456603852#¾:Õ,8 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4901053841920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139744349814784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4865476213924 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647233388453376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33970378 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784510588926429e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7331307 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚638604340227328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚642360214349824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17390627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33970378 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚642360197310464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33970376 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4900956968448ØHé2¾- _wandbruntime‚3762  _runtime‚3762ÉhoÑ ¿… /loss/FRt1/detection_loss‚0.5073449353280436 /loss/FRt2/detection_loss‚1.5982700422421114 1loss/SRqqt1/detection_loss‚0.7239084392070103 _step‚345 0loss/FRt1/assignment_loss‚1.7819008396667184 3loss/SRqqt1/assignment_loss‚0.47460686845505234 2loss/SRqqt2/assignment_loss‚0.5431634898869452 0loss/FBt1/detection_loss‚0.46593763663021975 epoch‚10 trainer/global_step‚4199 % _timestamp‚1.7845105889274616e+09 _runtime‚ 3762.50270666 0loss/FRt2/assignment_loss‚0.8692758825332676 /loss/FBt2/detection_loss‚0.7162363782623097 2loss/FBt1/assignment_loss‚0.003734934377007834 2loss/FBt2/assignment_loss‚0.020228429803549226 1loss/SRqqt2/detection_loss‚0.6609726889478482 %loss/total_loss‚8.365580565340084Ÿ*tNÀŠ /loss/FRt1/detection_loss‚0.5073449353280436 /loss/FRt2/detection_loss‚1.5982700422421114 1loss/SRqqt1/detection_loss‚0.7239084392070103 _step‚345 0loss/FRt1/assignment_loss‚1.7819008396667184 3loss/SRqqt1/assignment_loss‚0.47460686845505234 2loss/SRqqt2/assignment_loss‚0.5431634898869452 0loss/FBt1/detection_loss‚0.46593763663021975 epoch‚10 trainer/global_step‚4199 % _timestamp‚1.7845105889274616e+09 _runtime‚ 3762.50270666 0loss/FRt2/assignment_loss‚0.8692758825332676 /loss/FBt2/detection_loss‚0.7162363782623097 2loss/FBt1/assignment_loss‚0.003734934377007834 2loss/FBt2/assignment_loss‚0.020228429803549226 1loss/SRqqt2/detection_loss‚0.6609726889478482 %loss/total_loss‚8.365580565340084Ù¬®Qæ^Á‚(:Ó ÿðõÒþ˜Ë disk.rbd0.in‚0# proc.memory.rssMB‚ 2945.16015625 network.sent‚ 6.523929e+06 network.recv‚991445( disk./.usagePercent‚76.0044952029806 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚24305.33837890625 disk.md0.in‚0 disk.md0.out‚ 121.76171875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚11572.66455078125 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚24305.33837890625$ memory_percent‚95.49362659454346 cpu‚25.648626758527136 proc.cpu.threads‚17( disk.nvme1c1n1.out‚11572.66455078125 disk.dm-0.in‚ 0.00390625! disk.dm-0.out‚29443.380859375 disk.dm-1.in‚0( proc.memory.availableMB‚ 738.32421875* proc.memory.percent‚17.975831031799316% disk./.usageGB‚1166.8590126037598 disk.nvme1n1.in‚0 disk.dm-1.out‚ 6175.16015625¥Ók:“‚(:ˆ ÿðõÒ…ëÌ gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚281.892' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QíPUÂ(:Ê ŽñõÒµ¿ÈË network.sent‚ 6.588189e+06% disk./.usageGB‚1166.8591003417969( disk.nvme1c1n1.out‚11573.31787109375 disk.dm-0.in‚ 0.00390625 disk.dm-1.out‚ 6175.28515625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625) disk./.usagePercent‚76.00450091788288 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚11573.31787109375 disk.md0.in‚0 disk.md0.out‚ 122.140625 disk.nvme0n1.in‚ 0.00390625# disk.nvme0n1.out‚24306.75390625# proc.memory.rssMB‚ 2945.16015625 network.recv‚996353# disk.dm-0.out‚29444.51025390625 disk.dm-1.in‚0# memory_percent‚95.4904556274414 cpu‚25.647902427888535 disk.nvme1n1.in‚0% disk.nvme0c0n1.out‚24306.75390625% proc.memory.availableMB‚ 738.84375* proc.memory.percent‚17.975831031799316 proc.cpu.threads‚17ƒíÝ(žÄ‚(:“ ŽñõÒû¿ŒÌ gpu.0.gpu‚59.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.688' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.22933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Éïè^Å‚(:Ó ñõÒåäÀË disk.nvme1n1.in‚0% disk.nvme1n1.out‚11574.1064453125 disk.md0.out‚ 122.51953125 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 6.593127e+06 network.recv‚ 1.000161e+06% disk.nvme0c0n1.out‚24312.33203125' disk.nvme1c1n1.out‚11574.1064453125 cpu‚25.647255572291442# disk.nvme0n1.out‚24312.33203125 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚29449.98388671875" disk.dm-1.out‚6175.36767578125 disk.rbd0.out‚0% proc.memory.availableMB‚ 738.84375# proc.memory.rssMB‚ 2945.16015625) disk./.usagePercent‚75.98901626594369 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0# memory_percent‚95.4904556274414* proc.memory.percent‚17.975831031799316% disk./.usageGB‚1166.6213722229004 disk.nvme0n1.in‚ 0.00390625ˆ©žÆ‚(:“ ñõÒ…®Ì gpu.0.gpu‚48.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚284.952' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.98400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÝû²2Ç- _wandbruntime‚3805  _runtime‚3805U§Ÿ}ÇÈÁ & lr-AdamW/pg1‚0.0005578499688783568 & lr-AdamW/pg2‚0.0005578499688783568 trainer/global_step‚4249 % _timestamp‚1.7845106315360222e+09 _runtime‚3805.112247872 _step‚346Ê|!ÌÉÆ & lr-AdamW/pg2‚0.0005578499688783568 trainer/global_step‚4249 % _timestamp‚1.7845106315360222e+09 _runtime‚3805.112247872 _step‚346 & lr-AdamW/pg1‚0.0005578499688783568Úc)·Ì2Ê- _wandbruntime‚3805  _runtime‚3805ß½¹ûç'ËáO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚654552939590656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10172155 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17588136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146325950834176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚146325826952704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚4249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34352235 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚650748651573170 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42007609 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34352235 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4927291675136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42017752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34352233 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649625664954880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76369987 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚654552939590656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17588431 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚645828459972816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4919851039080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚654553298054144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34352233 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42017752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17588136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146325950834176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚146325826952704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 trainer/global_step‚4249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34352235 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚650748651573170 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42007609 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34352235 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4927291675136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42017752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34352233 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649625664954880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76369987 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚654552939590656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17588431 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚645828459972816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4919851039080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚654553298054144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34352233 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42017752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272Û˜èÝÖ2Í- _wandbruntime‚3805  _runtime‚3805€Lºð&ÎêM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146360125983232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚649773939457024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42017265 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4928743205376 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76377291 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34360026 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141403344706560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10174384 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17592394 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928401774080 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4921299988718 _runtime‚3805.734881353 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10174109 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4956781276672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17592116 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34360028 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4928743205376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34360028 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649773922417664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76387450 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7418010 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚4249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141403363318784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42027422 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76387450 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649773939457024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚654702682662400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚645975890640288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _step‚348 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7418007 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42027422 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚645975873600928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 % _timestamp‚1.7845106321590397e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34360026 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4928401774080 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76377291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146360241468928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0c¹­/õ&ÏïM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42027422 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76387450 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649773939457024 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚654702682662400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚645975890640288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _step‚348 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7418007 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42027422 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚645975873600928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 % _timestamp‚1.7845106321590397e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34360026 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4928401774080 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76377291 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146360241468928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146360125983232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚649773939457024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42017265 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4928743205376 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76377291 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34360026 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141403344706560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10174384 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17592394 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928401774080 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4921299988718 _runtime‚3805.734881353 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10174109 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4956781276672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17592116 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34360028 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4928743205376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34360028 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649773922417664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76387450 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7418010 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚4249 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141403363318784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184ÜgÀ.02Ð- _wandbruntime‚3805  _runtime‚3805K7¨ñ у trainer/global_step‚4249 % _timestamp‚1.7845106321606145e+09 2loss/SRqqt1/assignment_loss‚0.5429429950601774 2loss/SRqqt2/assignment_loss‚0.5370881121607765 3loss/FBt1/assignment_loss‚0.0026662441168272743 .loss/FBt2/detection_loss‚0.619566862554982 %loss/total_loss‚8.205643830886762 epoch‚10 _step‚349 0loss/FRt2/assignment_loss‚0.8444697734099537 4loss/FBt2/assignment_loss‚0.00015724895765272933 /loss/FRt1/detection_loss‚0.5037993396328849 1loss/SRqqt1/detection_loss‚0.7447383912256624 .loss/FRt2/detection_loss‚1.646975974769954 .loss/FBt1/detection_loss‚0.373689098334906 _runtime‚3805.735991975 /loss/FRt1/assignment_loss‚1.695803735620478 1loss/SRqqt2/detection_loss‚0.6937460550425074úÈŽkÒˆ 4loss/FBt2/assignment_loss‚0.00015724895765272933 /loss/FRt1/detection_loss‚0.5037993396328849 1loss/SRqqt1/detection_loss‚0.7447383912256624 .loss/FRt2/detection_loss‚1.646975974769954 .loss/FBt1/detection_loss‚0.373689098334906 _runtime‚3805.735991975 /loss/FRt1/assignment_loss‚1.695803735620478 1loss/SRqqt2/detection_loss‚0.6937460550425074 trainer/global_step‚4249 % _timestamp‚1.7845106321606145e+09 2loss/SRqqt1/assignment_loss‚0.5429429950601774 2loss/SRqqt2/assignment_loss‚0.5370881121607765 3loss/FBt1/assignment_loss‚0.0026662441168272743 .loss/FBt2/detection_loss‚0.619566862554982 %loss/total_loss‚8.205643830886762 epoch‚10 _step‚349 0loss/FRt2/assignment_loss‚0.8444697734099537Ýò—e^Ó‚(:Ó ¬ñõÒÐèË) disk./.usagePercent‚75.98902173237194 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚24314.03173828125% disk.nvme1c1n1.out‚11574.72265625* proc.memory.percent‚17.975831031799316 cpu‚25.646420762731978 network.sent‚ 6.59615e+06 network.recv‚ 1.003446e+06% disk./.usageGB‚1166.6214561462402 disk.dm-1.in‚0" disk.dm-1.out‚6175.40283203125# disk.nvme1n1.out‚11574.72265625# proc.memory.rssMB‚ 2945.16015625 disk.md0.out‚ 122.96875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚24314.03173828125 disk.nvme1c1n1.in‚0$ memory_percent‚95.49069404602051' proc.memory.availableMB‚ 738.8046875 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚29451.3134765625 disk.nvme1n1.in‚0 proc.cpu.threads‚17Ñ-²cžÔ‚(:“ ¬ñõÒ¹åÌ gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚265.057' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.35233333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇöÎœaÕ‚(:Ö »ñõҠƲË& proc.memory.availableMB‚ 738.984375# proc.memory.rssMB‚ 2945.16015625 cpu‚25.645651471711354 proc.cpu.threads‚17" disk.dm-0.out‚29467.5869140625( disk.nvme0c0n1.out‚24330.10302734375' disk.nvme1c1n1.out‚11575.7783203125% disk./.usageGB‚1166.6214027404785 disk.md0.out‚123.3125 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚24330.10302734375 network.recv‚ 1.008984e+06) disk./.usagePercent‚75.98901825373578 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ memory_percent‚95.48959732055664* proc.memory.percent‚17.975831031799316 network.sent‚ 6.659634e+06 disk.md0.in‚0" disk.dm-1.out‚6175.50439453125% disk.nvme1n1.out‚11575.7783203125±â5žÖ‚(:“ »ñõÒÔ–ˆÌ gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚203.252' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.75066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•tF²aׂ(:Ö ÊñõÒ“©¯Ë$ memory_percent‚96.99239730834961 network.recv‚ 1.012206e+06( disk.nvme0c0n1.out‚24481.64306640625& disk.nvme0n1.out‚24481.64306640625 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚11596.572265625 disk.md0.in‚0& proc.memory.availableMB‚ 492.765625* proc.memory.percent‚17.975831031799316) disk./.usagePercent‚76.00461919151248% disk./.usageGB‚1166.8609161376953 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 cpu‚25.644890786250325 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚11596.572265625 disk.md0.out‚ 123.5859375 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚29639.2197265625# proc.memory.rssMB‚ 2945.16015625 network.sent‚ 6.66525e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.dm-1.in‚0" disk.dm-1.out‚6175.60205078125ÿ‡¥žžØ‚(:“ ÊñõÒýØðË gpu.0.gpu‚67.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.004' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.33466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å9ä 2Ù- _wandbruntime‚3848  _runtime‚3848 fÅÚ¿ % _timestamp‚1.7845106751164858e+09 _runtime‚3848.693028944 _step‚350 % lr-AdamW/pg1‚0.000556417371936853 % lr-AdamW/pg2‚0.000556417371936853 trainer/global_step‚4299Ìùh{ÊÛÄ % lr-AdamW/pg2‚0.000556417371936853 trainer/global_step‚4299 % _timestamp‚1.7845106751164858e+09 _runtime‚3848.693028944 _step‚350 % lr-AdamW/pg1‚0.000556417371936853ÞY O2Ü- _wandbruntime‚3848  _runtime‚3848²dOº¢ÝáO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77244289 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚351 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4982860776960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34741883 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42492261 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚657039373022720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17789329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚658175668160178 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42502404 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4982860776960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34741885 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42492261 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7502973 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5011506727936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148040684668416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚653199993346416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4975674813762 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77234144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34741883 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4983202201088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚662022592263168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4975334252488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10286356 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚3848.693092614 N@DeviceStatsMonitor.on_train_batch_staÂŒ¨Ert/active_bytes.all.current‚ 358463488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚662022233799680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 % _timestamp‚1.7845106751171873e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17789627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10286651 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚657039390062080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143029196552704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143029177940480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚4299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚662022592263168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚657039373022720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5011611997184 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚662022233799680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4983202201088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚653199976307056 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42502404 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7502976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚657039390062080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34741883 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42492261 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚657039373022720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17789329 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚658175668160178 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42502404 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4982860776960 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34741885 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42492261 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7502973 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5011506727936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148040684668416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚653199993346416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4975674813762 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77234144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34741883 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4983202201088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚662022592263168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4975334252488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10286356 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚3848.693092614 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚662022233799680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 % _timestamp‚1.7845106751171873e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17789627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10286651 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚657039390062080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143029196552704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143029177940480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚4299 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚662022592263168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚657039373022720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5011611997184 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚662022233799680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4983202201088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚653199976307056 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496ßýܶ'2ß- _wandbruntime‚3849  _runtime‚3849'Æð&àêM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10288661 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77261752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7504707 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚662171618400768 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚658323849615268 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77251593 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 $ _timestamp‚1.784510675738078e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚662171976871424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17793368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34749678 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42501917 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657187647524864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚657187647524864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143062358210560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚4299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77251593 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148074980443136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4983970875904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3849.313883727 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34749678 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17793646 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚662171976871424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚653347406974528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42501917 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚658324207216014 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4976442640740 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4984312307200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5012727494656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5012622232576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77261752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42512074 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7504710 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚653347424013888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42512074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657187664564224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4984312307200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚352 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4976783202126 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚657187664564224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34749676À¬jMŠ!áïM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚662171976871424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚653347406974528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42501917 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚658324207216014 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4976442640740 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4984312307200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5012727494656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5012622232576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77261752 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42512074 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7504710 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚653347424013888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42512074 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657187664564224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4984312307200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚352 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4976783202126 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚657187664564224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34749676 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10288661 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77261752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7504707 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚662171618400768 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚658323849615268 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77251593 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 $ _timestamp‚1.784510675738078e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚662171976871424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17793368 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34749678 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42501917 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657187647524864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚657187647524864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143062358210560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚4299 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.èM3 kcurrent‚275 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77251593 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148074980443136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4983970875904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3849.313883727 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34749678 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17793646 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0à¹JK+2â- _wandbruntime‚3849  _runtime‚3849§Øÿ( ㇠/loss/FBt2/detection_loss‚0.6151613478149687 _step‚353 2loss/SRqqt2/assignment_loss‚0.5228945393216947 4loss/FBt2/assignment_loss‚0.00016759626572877552 /loss/FRt1/detection_loss‚0.4935977023817958 1loss/SRqqt1/detection_loss‚0.6189343505962328 $ _timestamp‚1.784510675739209e+09 _runtime‚3849.314542668 4loss/FBt1/assignment_loss‚4.5575620946047534e-05 0loss/SRqqt2/detection_loss‚0.764671101650452 %loss/total_loss‚8.037415826513293 epoch‚10 trainer/global_step‚4299 0loss/FRt2/assignment_loss‚0.8739714152841683 3loss/SRqqt1/assignment_loss‚0.48213366132444613 .loss/FRt2/detection_loss‚1.556659744769694 0loss/FRt1/assignment_loss‚1.6921962212279158 0loss/FBt1/detection_loss‚0.41698257025525026yØTÏäŒ 0loss/FRt1/assignment_loss‚1.6921962212279158 0loss/FBt1/detection_loss‚0.41698257025525026 /loss/FBt2/detection_loss‚0.6151613478149687 _step‚353 2loss/SRqqt2/assignment_loss‚0.5228945393216947 4loss/FBt2/assignment_loss‚0.00016759626572877552 /loss/FRt1/detection_loss‚0.4935977023817958 1loss/SRqqt1/detection_loss‚0.6189343505962328 $ _timestamp‚1.784510675739209e+09 _runtime‚3849.314542668 4loss/FBt1/assignment_loss‚4.5575620946047534e-05 0loss/SRqqt2/detection_loss‚0.764671101650452 %loss/total_loss‚8.037415826513293 epoch‚10 trainer/global_step‚4299 0loss/FRt2/assignment_loss‚0.8739714152841683 3loss/SRqqt1/assignment_loss‚0.48213366132444613 .loss/FRt2/detection_loss‚1.556659744769694á4E¬Ö_å‚(:Ô ÙñõÒΗË proc.cpu.threads‚17" disk.dm-0.out‚29715.4814453125 disk.nvme0n1.in‚ 0.00390625$ memory_percent‚96.99382781982422* proc.memory.percent‚17.976975440979004 network.sent‚ 6.668214e+06 network.recv‚ 1.015726e+06) disk./.usagePercent‚76.00449569992863 disk.dm-1.in‚0# proc.memory.rssMB‚ 2945.34765625 cpu‚25.64478405332001% disk./.usageGB‚1166.8590202331543 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚11631.7392578125 disk.md0.out‚123.9375 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚24523.67236328125& disk.nvme0n1.out‚24523.67236328125 disk.nvme1n1.in‚0% disk.nvme1n1.out‚11631.7392578125 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚6175.76708984375 disk.nvme0c0n1.in‚ 0.00390625% proc.memory.availableMB‚ 492.53125Ú>ížæ‚(:“ ÙñõÒ¸¸“Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚122.37' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.790000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C~fç‚(:Û èñõÒòÎÒË( disk.nvme0c0n1.out‚24524.92626953125$ memory_percent‚96.99552059173584( proc.memory.availableMB‚ 492.25390625* proc.memory.percent‚17.976975440979004 network.sent‚ 6.731905e+06 network.recv‚ 1.020678e+06 disk.nvme0n1.in‚ 0.00390625 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚24524.92626953125 disk.md0.out‚ 124.5078125# proc.memory.rssMB‚ 2945.34765625 cpu‚25.645312388670533 proc.cpu.threads‚17) disk./.usagePercent‚76.00453967982871% disk./.usageGB‚1166.8596954345703 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚11632.4580078125 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚29716.2080078125 disk.rbd0.in‚0' disk.nvme1c1n1.out‚11632.4580078125 disk.nvme1n1.in‚0" disk.dm-1.out‚6175.81005859375 disk.nvme0c0n1.in‚ 0.00390625ßO-æžè‚(:“ èñõÒÚï‘Ì gpu.0.gpu‚90.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚279.076' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.02533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š¼bXé‚(:Í ÷ñõÒÿÕ Ë disk.nvme0n1.in‚ 0.00390625 disk.md0.out‚ 125.22265625 disk.dm-1.in‚0 disk.dm-1.out‚6175.935546875 memory_percent‚ 96.9970703125# proc.memory.rssMB‚ 2945.34765625* proc.memory.percent‚17.976975440979004 cpu‚25.64616885747803) disk./.usagePercent‚76.00457496313838$ disk./.usageGB‚1166.860237121582( disk.nvme0c0n1.out‚24542.08447265625( disk.nvme1c1n1.out‚11633.81005859375" disk.dm-0.out‚29733.0771484375 proc.cpu.threads‚17 network.sent‚ 6.735633e+06 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625 proc.memory.availableMB‚492& disk.nvme1n1.out‚11633.81005859375 disk.md0.in‚0 network.recv‚ 1.023938e+06 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚24542.08447265625 disk.nvme1n1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0ç/Üžê‚(:“ ÷ñõÒ‰¢÷Ë gpu.0.gpu‚37.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚242.458' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚80.81933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š±K¯2ë- _wandbruntime‚3893  _runtime‚3893Íb1ÚÇìÁ & lr-AdamW/pg1‚0.0005549687711009503 & lr-AdamW/pg2‚0.0005549687711009503 trainer/global_step‚4349 % _timestamp‚1.7845107203760927e+09 _runtime‚3893.951844364 _step‚354ö–LôÌíÆ _runtime‚3893.951844364 _step‚354 & lr-AdamW/pg1‚0.0005549687711009503 & lr-AdamW/pg2‚0.0005549687711009503 trainer/global_step‚4349 % _timestamp‚1.7845107203760927e+09â(àp2î- _wandbruntime‚3893  _runtime‚3893W¶x‚ç'ïáO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚669491886472192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42987056 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚669491886472192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7589673 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5067316511232 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚669491528008704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42976913 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7589676 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚660571526720016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚664453098129920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚665602684747186 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5038771302912 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5030817465896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5031158027170 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17990856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149755638128640 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚665602327146552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚355 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚669491528008704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10400903 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149755507955712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚664453115169280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 % _timestamp‚1.7845107203802016e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚3893.955920381 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚664453115169280 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35131535 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17990576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 trainer/global_step‚4349 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5038429878784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚664453098129920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10401180 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5067428071936 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚669491528008704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42976913 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7589676 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚660571526720016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚664453098129920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚665602684747186 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5038771302912 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5030817465896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5031158027170 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17990856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149755638128640 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚665602327146552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚355 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚669491528008704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10400903 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149755507955712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚664453115169280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 % _timestamp‚1.7845107203802016e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚3893.955920381 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚664453115169280 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35131535 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17990576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 trainer/global_step‚4349 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5038429878784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚664453098129920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10401180 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5067428071936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7589673 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85ã¾kò,2ñ- _wandbruntime‚3894  _runtime‚3894ÛgÀÓòëM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78125895 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42986569 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10403450 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42996726 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5039539977728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35139328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42986569 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5032266415534 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚669641271080448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚669641271080448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5039881409024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42996726 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149789929406976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78136054 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5039539977728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149789803435520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845107210082152e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35139326 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78125895 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5031925854148 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.cu0ëGyrrent‚276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10403177 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5068431720960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144721371714560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚356 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78136054 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144721390326784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚664601372632064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚665750866202276 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17994860 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚660718940348128 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17994584 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚664601389671424 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42986569 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10403450 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42996726 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5039539977728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35139328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42986569 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5032266415534 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚669641271080448 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚669641271080448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5039881409024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42996726 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149789929406976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78136054 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5039539977728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149789803435520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845107210082152e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35139326 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78125895 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5031925854148 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10403177 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5068431720960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144721371714560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚356 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78136054 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144721390326784 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚664601372632064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚665750866202276 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17994860 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚660718940348128 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17994584 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚664601389671424 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3938.795136926 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35521185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚676961180681216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚671866823237120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5093998980608 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7676373 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146347204948480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18192440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5086641240578 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚151470337058304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚673029701334194 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚359 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚671866823237120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43471708 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43471708 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚673029343733560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43461565 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5094340404736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚676960822217728 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚151470463036928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43461565 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35521185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚671866840276480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚667943043054256 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5086300679304 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚4399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35521183 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚676960822217728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5123239476224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5123132109824w·$[‚åO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5094340404736 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚676961180681216 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7676376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚671866823237120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43471708 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ÉøÒ)"43471708 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚673029343733560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43461565 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5094340404736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚676960822217728 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚151470463036928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43461565 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35521185 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚671866840276480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚667943043054256 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5086300679304 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚4399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35521183 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚676960822217728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5123239476224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5123132109824 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78992893 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18192157 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3938.795136926 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35521185 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚676961180681216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚671866823237120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5093998980608 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7676373 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146347204948480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18192440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5086641240578 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚151470337058304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚673029701334194 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚359 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79010356 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚151504629687296 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79000197 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚672015114778624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43471221 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚673177882789284 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚677110565289472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4399 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35528976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43481378 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5095450510848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚146380403830784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚672015097739264 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3939.421223515 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35528976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10518365 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5095109079552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10518083 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18196190 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5087749628942 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚677110565289472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚668090490761088 _step‚360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35528978 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35528978 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚673178240390030 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚677110206818816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79010356 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7845107658454862e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672015114778624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚151504753561600 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5095450510848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7678107 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5095109079552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3939.421223515 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35528976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10518365 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5095109079552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10518083 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18196190 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5087749628942 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚677110565289472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚668090490761088 _step‚360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35528978 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35528978 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚673178240390030 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚677110206818816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79010356 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7845107658454862e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672015114778624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚151504753561600 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5095450510848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7678107 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5095109079552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79010356 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚151504629687296 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79000197 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚672015114778624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43471221 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚673177882789284 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚677110565289472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4399 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35528976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43481378 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5095450510848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚146380403830784èË'“2†- _wandbruntime‚3939  _runtime‚3939_Ú‡€ /loss/FRt2/assignment_loss‚0.783652826566631 2loss/SRqqt2/assignment_loss‚0.6074760810502009 .loss/FRt2/detection_loss‚1.627307465350879 1loss/SRqqt1/detection_loss‚0.8073216014766668 /loss/FBt1/detection_loss‚0.4433239263174965 %loss/total_loss‚8.419521536157248 epoch‚10 trainer/global_step‚4399 _runtime‚3939.421802076 _step‚361 0loss/FRt1/assignment_loss‚1.8293329034515513 1loss/FBt2/assignment_loss‚0.03704151642532214 /loss/FBt2/detection_loss‚0.5206201154075257 % _timestamp‚1.7845107658465428e+09 2loss/SRqqt1/assignment_loss‚0.5679685346619684 1loss/FBt1/assignment_loss‚0.06434143190917248 /loss/FRt1/detection_loss‚0.4630230958884116 1loss/SRqqt2/detection_loss‚0.6681120376514206]5/ ˆ… /loss/FBt1/detection_loss‚0.4433239263174965 %loss/total_loss‚8.419521536157248 epoch‚10 trainer/global_step‚4399 _runtime‚3939.421802076 _step‚361 0loss/FRt1/assignment_loss‚1.8293329034515513 1loss/FBt2/assignment_loss‚0.03704151642532214 /loss/FBt2/detection_loss‚0.5206201154075257 % _timestamp‚1.7845107658465428e+09 2loss/SRqqt1/assignment_loss‚0.5679685346619684 1loss/FBt1/assignment_loss‚0.06434143190917248 /loss/FRt1/detection_loss‚0.4630230958884116 1loss/SRqqt2/detection_loss‚0.6681120376514206 /loss/FRt2/assignment_loss‚0.783652826566631 2loss/SRqqt2/assignment_loss‚0.6074760810502009 .loss/FRt2/detection_loss‚1.627307465350879 1loss/SRqqt1/detection_loss‚0.8073216014766668é¨A X[‰‚(:Ð ³òõÒã®îË network.sent‚ 6.824528e+06 disk.nvme1c1n1.in‚0 disk.md0.in‚0# memory_percent‚97.0792293548584# proc.memory.rssMB‚ 2959.83203125 cpu‚25.65177248058215 proc.cpu.threads‚17 network.recv‚ 1.039139e+06% disk./.usageGB‚1166.6225395202637 disk.nvme0n1.in‚ 0.00390625# disk.nvme0n1.out‚24551.57421875 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚11636.685546875' proc.memory.availableMB‚ 478.5390625) disk./.usagePercent‚75.98909229899128 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚29741.4208984375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625* proc.memory.percent‚18.065381050109863 disk.dm-1.in‚0! disk.dm-1.out‚6176.2978515625 disk.rbd0.in‚0% disk.nvme0c0n1.out‚24551.57421875& disk.nvme1c1n1.out‚11636.685546875 disk.md0.out‚ 126.93359375NšÆ»“Š‚(:ˆ ³òõÒýï©Ì gpu.0.gpu‚80.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.155' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.385 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|ð,¡2‹- _wandbruntime‚3956  _runtime‚3956‹ÆQ§Œëz =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17227284610271454 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8470332622528076 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4772147238254547 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6416897773742676 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6418662071228027 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14666667580604553 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7130892872810364 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978691339492798 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9150725603103638 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1679755449295044 ("Purity/*FRt2SRqqt1FBt/ev³Œ£Ê9ent_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 *jet/accuracy_1_of_4‚0.9708340764045715 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8169416785240173 +particle/sensitivity‚0.6385762095451355 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9342657923698425 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8417422771453857 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7106215953826904 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.35395821928977966 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018241018056869507 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6352737545967102 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965729713439941 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6680771112442017 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18570400774478912 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 _step‚362 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5990008115768433 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9985337257385254 particle/accuracy_3_of_3‚1 +particle/specificity‚0.9044610857963562 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15397204458713531 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8255857825279236 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8169416785240173 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,jet/accuracy_4_of_4‚0.009931244887411594 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43011441826820374 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4655199646949768 particle/accuracy_1_of_1‚1 jet/accuracy_5_of_5‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15703336894512177 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20007121562957764 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8473578691482544 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 particle/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7066453695297241 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2930046617984772 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7166485786437988 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7394869327545166 +jet/accuracy_3_of_4‚0.08728202432394028 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2545512616634369 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13047932088375092 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.240444615483284 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6180508732795715 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9874510765075684 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 *jet/accuracy_1_of_1‚0.3090619444847107 % _timestamp‚1.7845107826733112e+09 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6712406873703003 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9882786870002747 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6605548858642578 +"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.75 /particle/accuracy_4_of_4‚0.9594709277153015 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6212307810783386 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949473142623901 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4861111044883728 *jet/accuracy_2_of_4‚0.7788895964622498 trainer/global_step‚4399 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4325592815876007 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.672447681427002 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5553838014602661 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0171380415558815 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17478196322917938 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18049031496047974 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7366705536842346 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03659024089574814 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912068247795105 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9931187629699707 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958221912384033 particle/accuracy_2_of_3‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.35648149251937866 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7759051322937012 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6396759748458862 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16566242277622223 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16380012035369873 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06618773937225342 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3645833432674408 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0254363976418972 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15365701913833618 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8414210081100464 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13039477169513702 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19825215637683868 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2871982157230377 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18692152202129364 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6667097806930542 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6418662071228027 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17227284610271454 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.75 jet/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949354529380798 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_2_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1851380169391632 particle/accuracy_2_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958750605583191 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16716916859149933 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5516666173934937 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_2_of_2‚0.14402861893177032 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11413517594337463 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6225350499153137 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18587476015090942 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06618773937225342 'particle/f_score‚0.6424285173416138 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7212545275688171 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7401894927024841 *jet/accuracy_2_of_3‚0.5380070209503174 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _runtime‚3956.249393339 )loss/val_total_loss‚8.078749662205075 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 particle/accuracy_1_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7348194718360901 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6396445035934448 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17294704914093018 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8386244177818298 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8603647947311401 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 *jet/accuracy_1_of_2‚0.7714405059814453 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37688276171684265 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5424017310142517 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.763454258441925 particle/accuracy_2_of_2‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1383962333202362 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05679631978273392 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4706669747829437 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 epoch‚10 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6813905835151672 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7175354957580566 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8727024793624878 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6788411140441895 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14207860827445984 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 jet/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19452211260795593 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5963192582130432 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05081837624311447 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03174668923020363 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5527715086936951 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965227246284485 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945749640464783 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.07128240168094635 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8933156132698059 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6225271224975586 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010130483657121658 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7375038266181946 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12122726440429688 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.057188209146261215 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7111991047859192 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37479159235954285 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7735360264778137 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6524325609207153 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3828125 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6449908018112183 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7509424090385437 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44536298513412476 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05028765648603439 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 šv;v=ðz 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11413517594337463 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6225350499153137 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18587476015090942 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06618773937225342 'particle/f_score‚0.6424285173416138 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7212545275688171 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7401894927024841 *jet/accuracy_2_of_3‚0.5380070209503174 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _runtime‚3956.249393339 )loss/val_total_loss‚8.078749662205075 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 particle/accuracy_1_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7348194718360901 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6396445035934448 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17294704914093018 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8386244177818298 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8603647947311401 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 *jet/accuracy_1_of_2‚0.7714405059814453 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37688276171684265 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5424017310142517 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.763454258441925 particle/accuracy_2_of_2‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1383962333202362 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05679631978273392 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4706669747829437 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 epoch‚10 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6813905835151672 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7175354957580566 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8727024793624878 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6788411140441895 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14207860827445984 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 jet/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19452211260795593 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5963192582130432 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05081837624311447 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03174668923020363 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5527715086936951 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965227246284485 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945749640464783 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.07128240168094635 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8933156132698059 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6225271224975586 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010130483657121658 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7375038266181946 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12122726440429688 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.057188209146261215 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7111991047859192 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37479159235954285 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7735360264778137 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6524325609207153 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3828125 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6449908018112183 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7509424090385437 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44536298513412476 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05028765648603439 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17227284610271454 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8470332622528076 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4772147238254547 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6416897773742676 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6418662071228027 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14666667580604553 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7130892872810364 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9978691339492798 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9150725603103638 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1679755449295044 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 *jet/accuracy_1_of_4‚0.9708340764045715 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8169416785240173 +particle/sensitivity‚0.6385762095451355 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9342657923698425 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8417422771453857 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7106215953826904 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.35395821928977966 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018241018056869507 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6352737545967102 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965729713439941 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6680771112442017 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18570400774478912 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 _step‚362 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5990008115768433 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9985337257385254 particle/accuracy_3_of_3‚1 +particle/specificity‚0.9044610857963562 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15397204458713531 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8255857825279236 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8169416785240173 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,jet/accuracy_4_of_4‚0.009931244887411594 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43011441826820374 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4655199646949768 particle/accuracy_1_of_1‚1 jet/accuracy_5_of_5‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15703336894512177 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20007121562957764 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8473578691482544 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 particle/accuracy_3_of_5‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7066453695297241 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2930046617984772 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7166485786437988 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7394869327545166 +jet/accuracy_3_of_4‚0.08728202432394028 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2545512616634369 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13047932088375092 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.240444615483284 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6180508732795715 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9874510765075684 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 *jet/accuracy_1_of_1‚0.3090619444847107 % _timestamp‚1.7845107826733112e+09 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6712406873703003 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9882786870002747 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6605548858642578 +"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.75 /particle/accuracy_4_of_4‚0.9594709277153015 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6212307810783386 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949473142623901 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4861111044883728 *jet/accuracy_2_of_4‚0.7788895964622498 trainer/global_step‚4399 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4325592815876007 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.672447681427002 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5553838014602661 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0171380415558815 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17478196322917938 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18049031496047974 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7366705536842346 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03659024089574814 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912068247795105 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9931187629699707 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958221912384033 particle/accuracy_2_of_3‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.35648149251937866 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7759051322937012 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6396759748458862 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16566242277622223 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16380012035369873 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06618773937225342 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3645833432674408 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0254363976418972 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15365701913833618 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8414210081100464 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13039477169513702 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19825215637683868 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2871982157230377 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18692152202129364 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6667097806930542 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6418662071228027 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17227284610271454 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.75 jet/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949354529380798 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_2_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1851380169391632 particle/accuracy_2_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958750605583191 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16716916859149933 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5516666173934937 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_2_of_2‚0.14402861893177032êÛòþª`Ž‚(:Õ ÂòõÒŠ¸Ë disk.rbd0.out‚0( disk.nvme0c0n1.out‚24552.56201171875 network.sent‚ 6.874048e+06) disk./.usagePercent‚75.97360864094813 disk.md0.out‚ 127.19140625! disk.dm-1.out‚6176.3955078125 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚24552.56201171875 network.recv‚ 1.044032e+06 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚11637.32470703125' proc.memory.availableMB‚ 462.8671875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625% disk./.usageGB‚1166.3848266601562 disk.md0.in‚0 disk.dm-0.out‚ 29742.375 disk.dm-1.in‚0 disk.nvme1n1.in‚0& disk.nvme1n1.out‚11637.32470703125 disk.dm-0.in‚ 0.00390625$ memory_percent‚97.17488288879395# proc.memory.rssMB‚ 2975.63671875* proc.memory.percent‚18.161845207214355 cpu‚25.6583732815902449‰çÑ‚(:† ÂòõÒ°‰¨Ì gpu.0.gpu‚2.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.778' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n­Ø‘‚(:† ÑòõÒ€›¯Ë gpu.0.gpu‚41.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.94' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.98 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ+CÚ‘‚(:Ö ÑòõÒºÂøË network.sent‚ 6.932879e+06 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚ 29755.53125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme1n1.out‚11639.4853515625$ memory_percent‚97.17488288879395 proc.cpu.threads‚17) disk./.usagePercent‚75.97359696266959" disk.dm-1.out‚6176.57763671875 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 2975.63671875( disk.nvme0c0n1.out‚24564.46240234375 network.recv‚ 1.049305e+06ãÛš¡M disk.md0.in‚0 disk.md0.out‚ 127.51953125 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚24564.46240234375' proc.memory.availableMB‚ 462.8671875* proc.memory.percent‚18.161845207214355% disk./.usageGB‚1166.3846473693848 disk.dm-1.in‚0' disk.nvme1c1n1.out‚11639.4853515625 cpu‚25.657520726001046 Z&Ï`’‚(:Õ àòõÒÝ…ËË disk.md0.out‚ 127.90625 disk.dm-1.in‚0 network.sent‚ 6.937672e+06 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme0n1.out‚24565.2822265625( disk.nvme1c1n1.out‚11640.05517578125 disk.nvme1n1.in‚0" disk.dm-1.out‚6176.64794921875# proc.memory.rssMB‚ 2975.68359375 cpu‚25.656561296207002 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚11640.05517578125 disk.md0.in‚0$ memory_percent‚97.17535972595215' proc.memory.availableMB‚ 462.7890625 proc.cpu.threads‚17 network.recv‚ 1.052669e+06) disk./.usagePercent‚75.97360019283174 disk.rbd0.out‚0 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚ 29756.0078125* proc.memory.percent‚18.162131309509277% disk./.usageGB‚1166.3846969604492' disk.nvme0c0n1.out‚24565.2822265625 disk.nvme0n1.in‚ 0.00390625*yl›“‚(: àòõÒ‚ö„Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.14' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.04666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•¯€õ2”- _wandbruntime‚3999  _runtime‚3999){D'Ç•Á _step‚363 & lr-AdamW/pg1‚0.0005520239309315721 & lr-AdamW/pg2‚0.0005520239309315721 trainer/global_step‚4449 % _timestamp‚1.7845108259493947e+09 _runtime‚3999.525809511«ÆÿÌ–Æ & lr-AdamW/pg1‚0.0005520239309315721 & lr-AdamW/pg2‚0.0005520239309315721 trainer/global_step‚4449 % _timestamp‚1.7845108259493947e+09 _runtime‚3999.525809511 _step‚363ëáØ¿2—- _wandbruntime‚3999  _runtime‚3999èúâÀæ'˜àO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚682397920177689 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚677235849475440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚686393136601600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5169885918208 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686392778138112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚681222909259264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18432136 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚686393136601600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44027711 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44037854 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚681222892219904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5199834825728 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚686392778138112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5199729556480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚682398277778323 $ _timestamp‚1.784510825950215e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚364 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36033643 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360œç,ë'™åO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚686393136601600 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5169885918208 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686392778138112 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚681222909259264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18432136 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚686393136601600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44027711 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44037854 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚681222892219904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5199834825728 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚686392778138112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5199729556480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚682398277778323 $ _timestamp‚1.784510825950215e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚364 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36033643 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚682397920177689 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚677235849475440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452ìÊ‘ a2š- _wandbruntime‚4000  _runtime‚4000€Îð&›êM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36041434 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚4000.153735225 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚686542521209856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18436404 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚677383280142912 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚677383263103552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚686542521209856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5200949125120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80088960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5170996017152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80088960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚365 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7768926 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36041436 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44047524 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚681371166722048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18436122 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7768923 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5200850154496 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚686542162739200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153262827539456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚681371166722048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚681371183761408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44037367 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10667478 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5171337448448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148061995997184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5170996017152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44037367 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚682546459233413 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚677383280142912 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚677383263103552 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚686542521209856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5200949125120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80088960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5170996017152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80088960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚365 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7768926 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36041436 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44047524 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚681371166722048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18436122 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7768923 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5200850154496 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚686542162739200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153262827539456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚681371166722048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚681371183761408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44037367 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10667478 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5171337448448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚148061995997184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5170996017152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44037367 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚682546459233413 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336í1Pa{2- _wandbruntime‚4000  _runtime‚4000E~'ž€ 1loss/SRqqt2/detection_loss‚0.7039144689655643 _step‚366 0loss/FRt1/assignment_loss‚1.8572419071974962 .loss/FRt2/detection_loss‚1.524299124311856 /loss/FBt1/detection_loss‚0.5313505907488604 %loss/total_loss‚8.430155991862492 % _timestamp‚1.7845108265793402e+09 _runtime‚4000.154719307 0loss/FRt2/assignment_loss‚0.8043266628378181 2loss/SRqqt1/assignment_loss‚0.5561208967660884 0loss/SRqqt2/assignment_loss‚0.59063958854851 1loss/FBt1/assignment_loss‚0.02895337807718434 2loss/FBt2/assignment_loss‚0.009106039606716906 /loss/FRt1/detection_loss‚0.5455937660849611 1loss/SRqqt1/detection_loss‚0.7669685810775752 /loss/FBt2/detection_loss‚0.5116409876398624 epoch‚11 trainer/global_step‚4449ÌG^æ Ÿ… 1loss/SRqqt2/detection_loss‚0.7039144689655643 _step‚366 0loss/FRt1/assignment_loss‚1.8572419071974962 .loss/FRt2/detection_loss‚1.524299124311856 /loss/FBt1/detection_loss‚0.5313505907488604 %loss/total_loss‚8.430155991862492 % _timestamp‚1.7845108265793402e+09 _runtime‚4000.154719307 0loss/FRt2/assignment_loss‚0.8043266628378181 2loss/SRqqt1/assignment_loss‚0.5561208967660884 0loss/SRqqt2/assignment_loss‚0.59063958854851 1loss/FBt1/assignment_loss‚0.02895337807718434 2loss/FBt2/assignment_loss‚0.009106039606716906 /loss/FRt1/detection_loss‚0.5455937660849611 1loss/SRqqt1/detection_loss‚0.7669685810775752 /loss/FBt2/detection_loss‚0.5116409876398624 epoch‚11 trainer/global_step‚4449îö6˜©^ ‚(:Ó ïòõÒ¬çÆË! proc.memory.rssMB‚ 2975.734375$ disk.nvme1n1.out‚11641.771484375 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625 proc.cpu.threads‚17 disk.md0.out‚ 128.46875 disk.rbd0.in‚0& disk.nvme0n1.out‚24576.26513671875) proc.memory.percent‚18.16244125366211 cpu‚25.655791245080646 network.recv‚ 1.056195e+06) disk./.usagePercent‚75.97360168367581% disk./.usageGB‚1166.3847198486328& disk.nvme1c1n1.out‚11641.771484375! disk.dm-0.out‚29767.291015625 disk.dm-1.in‚0" disk.dm-1.out‚6176.85986328125 disk.nvme0n1.in‚ 0.00390625 network.sent‚ 6.940636e+06 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚24576.26513671875$ memory_percent‚97.17628955841064( proc.memory.availableMB‚ 462.63671875Ÿ‰‚¶œ¡‚(:‘ ïòõÒ˜ïÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.876' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.95866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù{aÄ[¢‚(:Ð þòõÒ“ß®Ë disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0" disk.nvme1n1.out‚ 11642.4921875) proc.memory.percent‚18.16244125366211 disk.dm-1.in‚0" disk.dm-1.out‚6176.95361328125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚24577.38818359375' proc.memory.availableMB‚ 462.6328125 network.sent‚ 7.004041e+06( disk./.usagePercent‚75.9735994474097 disk.dm-0.in‚ 0.00390625$ disk.nvme1c1n1.out‚ 11642.4921875 disk.md0.in‚0 disk.md0.out‚ 128.98828125$ memory_percent‚97.17631340026855! proc.memory.rssMB‚ 2975.734375 network.recv‚ 1.061197e+06! disk.dm-0.out‚29767.935546875 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚24577.38818359375 disk.nvme1n1.in‚0 cpu‚25.655028947157938 proc.cpu.threads‚17% disk./.usageGB‚1166.3846855163574 disk.rbd0.out‚0üüùž£‚(:“ þòõÒýƒÌ gpu.0.gpu‚93.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚122.981' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.99366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýZ‘^¤‚(:Ó óõÒ¢ŸÜË) disk./.usagePercent‚76.00460453154578 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚6177.09716796875' disk.nvme1c1n1.out‚11723.4423828125 disk.md0.in‚0) proc.memory.percent‚18.16244125366211 proc.cpu.threads‚17 network.sent‚ 7.009793e+06 network.recv‚ 1.06484e+06 disk.rbd0.out‚0( disk.nvme0c0n1.out‚24825.42822265625 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚24825.42822265625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0" disk.dm-0.out‚30095.7392578125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme1n1.out‚11723.4423828125 disk.md0.out‚ 129.48046875$ memory_percent‚97.17636108398438# proc.memory.availableMB‚462.625 cpu‚25.654326793902257% disk./.usageGB‚1166.8606910705566 disk.dm-1.in‚0! proc.memory.rssMB‚ 2975.734375׌bƒž¥‚(:“ óõÒ¿êàÎ gpu.0.gpu‚48.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚124.565' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.52166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bÆ»Ý2¦- _wandbruntime‚4044  _runtime‚4044ȯæǧÁ % _timestamp‚1.7845108705034819e+09 _runtime‚4044.080023629 _step‚367 & lr-AdamW/pg1‚0.0005505278807396776 & lr-AdamW/pg2‚0.0005505278807396776 trainer/global_step‚4499}j=WÌ¨Æ _step‚367 & lr-AdamW/pg1‚0.0005505278807396776 & lr-AdamW/pg2‚0.0005505278807396776 trainer/global_step‚4499 % _timestamp‚1.7845108705034819e+09 _runtime‚4044.080023629ï‡j6f2©- _wandbruntime‚4044  _runtime‚4044dã+ç'ªáO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44512363 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚4499 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5255693561344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5255592486400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚693862430810624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚684607365809680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36423293 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚684607382849040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44522506 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18633675 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154943522792448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚688636634366464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36423293 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5217570955017 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 _step‚368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚693862072347136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5225455020032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80945799 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44522506 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5225455020032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7853889 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚689825294365331 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚688636617327104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5225796444160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚4044.080102899 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845108705041945e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚693862072347136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚688636634366464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36423291 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18633372 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0â ˜«æO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚689825294365331 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚688636617327104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5225796444160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚4044.080102899 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845108705041945e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚693862072347136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚688636634366464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36423291 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18633372 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44512363 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_spliU•hÑç t_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚4499 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5255693561344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5255592486400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚693862430810624 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚684607365809680 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36423293 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚684607382849040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44522506 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18633675 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154943522792448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚688636634366464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36423293 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5217570955017 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 _step‚368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚693862072347136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5225455020032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80945799 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44522506 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5225455020032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7853889ðó¹2¬- _wandbruntime‚4044  _runtime‚4044 7Påð&­êM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚4499 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80953103 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36431084 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18637660 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18637381 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚689973475820421 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36431086 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5256708253184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚694011456948224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36431084 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5226906550272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _step‚369 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7855623 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5219019904655 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4044.708931482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚688784908868608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5226906550272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44532176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154977823016448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5218679343269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44522019 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10781758 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7855626 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149721009501184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80963262 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚688784891829248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80963262 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚694011815418880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36431086 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80953103 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5226565118976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5256813515264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚688784891829248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚684754796477152 $ _timestamp‚1.784510871133084e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44532176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18637381 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚689973475820421 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36431086 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5256708253184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚694011456948224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36431084 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5226906550272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _step‚369 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7855623 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5219019904655 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4044.708931482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚688784908868608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5226906550272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44532176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154977823016448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5218679343269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44522019 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10781758 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7855626 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149721009501184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80963262 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚688784891829248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80963262 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚694011815418880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36431086 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80953103 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5226565118976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5256813515264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚688784891829248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚684754796477152 $ _timestamp‚1.784510871133084e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44532176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10893676 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845109141445477e+09 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚372 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5281365545984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18834544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚696050342434304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701331366556160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚701331725019648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7940589 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚696050342434304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5281024121856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚691978916222640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156658453509632³¢e—ë'½åO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18834265 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚701331725019648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5281024121856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _runtime‚ 4087.72169381 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10893676 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845109141445477e+09 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚372 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5281365545984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18834544 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚696050342434304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701331366556160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚701331725019648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7940589 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚696050342434304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5281024121856 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚691978916222640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156658453509632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5311509699584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7940592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚696050359473664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36812941 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5273394729699 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5281365545984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81820101 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚697252310952339 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44997015 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18838256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5274162556677 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156692751307776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151380023005184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81837564 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81837564 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10896209 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚697400850008175 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚697400492407429 trainer/global_step‚4549 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156692631627776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36820736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81827405 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7942326 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚701481109627904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36820736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18838535 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚701480751157248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45016828 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5282475652096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5282134220800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚696198633975808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚701481109627904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10895933 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45016828 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5312627234816 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36820734 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚692126346890112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚696198616936448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5312728302592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184Kšeµ<ÀïM EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚696198633975808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚701481109627904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10895933 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45016828 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5312627234816 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36820734 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚692126346890112 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚696198616936448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5312728302592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18838256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5274162556677 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156692751307776 cODeviceStatsMonitor.DŸQ¹on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151380023005184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81837564 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81837564 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10896209 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚697400850008175 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚697400492407429 trainer/global_step‚4549 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156692631627776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36820736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81827405 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7942326 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚701481109627904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36820736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18838535 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚701480751157248 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45016828 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5282475652096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5282134220800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0õñc2Á- _wandbruntime‚4088  _runtime‚4088ŸWŽ ‡ 2loss/SRqqt1/assignment_loss‚0.5424184216100117 /loss/FRt1/detection_loss‚0.5266908213943438 1loss/SRqqt2/detection_loss‚0.7267753856701722 0loss/FBt2/detection_loss‚0.47068054486797856 epoch‚11 _runtime‚4088.337061246 0loss/FRt2/assignment_loss‚0.7970638749308122 3loss/FBt2/assignment_loss‚0.0025397116248606316 %loss/total_loss‚8.416191142753856 trainer/global_step‚4549 % _timestamp‚1.7845109147603738e+09 _step‚374 /loss/FRt2/detection_loss‚1.6274294177699657 1loss/SRqqt1/detection_loss‚0.7609530041659985 2loss/SRqqt2/assignment_loss‚0.5984093519667594 3loss/FBt1/assignment_loss‚0.0022867422951738142 /loss/FBt1/detection_loss‚0.5681666479535384 0loss/FRt1/assignment_loss‚1.7927772185042417r¾èõÃŒ 0loss/FRt1/assignment_loss‚1.7927772185042417 2loss/SRqqt1/assignment_loss‚0.5424184216100117 /loss/FRt1/detection_loss‚0.5266908213943438 1loss/SRqqt2/detection_loss‚0.7267753856701722 0loss/FBt2/detection_loss‚0.47068054486797856 epoch‚11 _runtime‚4088.337061246 0loss/FRt2/assignment_loss‚0.7970638749308122 3loss/FBt2/assignment_loss‚0.0025397116248606316 %loss/total_loss‚8.416191142753856 trainer/global_step‚4549 % _timestamp‚1.7845109147603738e+09 _step‚374 /loss/FRt2/detection_loss‚1.6274294177699657 1loss/SRqqt1/detection_loss‚0.7609530041659985 2loss/SRqqt2/assignment_loss‚0.5984093519667594 3loss/FBt1/assignment_loss‚0.0022867422951738142 /loss/FBt1/detection_loss‚0.5681666479535384ö+gÖ_Ä‚(:Ô ÉóõÒ°¨¤Ë$ memory_percent‚97.17748165130615( proc.memory.availableMB‚ 462.44140625) proc.memory.percent‚18.16272735595703 cpu‚25.653418044528607 disk.nvme0c0n1.in‚ 0.00390625 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625! disk.dm-0.out‚30284.287109375 disk.dm-1.out‚6177.494140625 proc.memory.rssMB‚ 2975.78125 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚24906.18115234375 disk.nvme1n1.in‚0 proc.cpu.threads‚17 network.sent‚ 7.085783e+06 network.recv‚ 1.0798e+06) disk./.usagePercent‚76.00458987157909$ disk./.usageGB‚1166.860466003418 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚24906.18115234375& disk.nvme1n1.out‚11834.16943359375 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚11834.16943359375 disk.md0.out‚ 130.63671875 disk.dm-1.in‚0¬$éžÅ‚(:“ ÉóõÒÁ‚ÓË gpu.0.gpu‚63.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.799' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.93299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Rušš_Æ‚(:Ô ØóõÒ¶”ÂË disk.md0.out‚ 130.9921875( proc.memory.availableMB‚ 464.01953125 proc.memory.rssMB‚ 2975.78125 network.sent‚ 7.147286e+06 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme0n1.out‚24909.107421875( disk.nvme1c1n1.out‚11834.85791015625 disk.md0.in‚0 disk.rbd0.in‚0 cpu‚25.652796696694143 disk.nvme0n1.in‚ 0.00390625 disk.dm-0.in‚ 0.00390625 disk.dm-1.out‚6177.529296875 disk.rbd0.out‚0$ memory_percent‚97.16784954071045) proc.memory.percent‚18.16272735595703 proc.cpu.threads‚17 network.recv‚ 1.084911e+06% disk./.usageGB‚1166.8604850769043# disk.dm-0.out‚30287.09912109375 disk.dm-1.in‚0) disk./.usagePercent‚76.00459111394915& disk.nvme0c0n1.out‚24909.107421875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0& disk.nvme1n1.out‚11834.85791015625jœÇ‚(:‘ ØóõÒå²ÿË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.604' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.20133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³FcÈ‚(:Ø çóõÒñ½ÍË( disk.nvme1c1n1.out‚11835.54833984375 network.recv‚ 1.088318e+06% disk./.usageGB‚1166.8609199523926 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚30287.90380859375 cpu‚25.65224210929465) disk./.usagePercent‚76.00461943998648 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme1n1.out‚11835.54833984375 disk.md0.in‚0' proc.memory.availableMB‚ 465.7109375* proc.memory.percent‚18.163037300109863 disk.nvme1n1.in‚0 network.sent‚ 7.152094e+06 disk.rbd0.in‚0& disk.nvme0c0n1.out‚24910.357421875 disk.nvme0n1.in‚ 0.00390625$ disk.nvme0n1.out‚24910.357421875 disk.dm-1.in‚0$ memory_percent‚97.15752601623535 disk.md0.out‚ 131.48046875! disk.dm-1.out‚6177.6103515625# proc.memory.rssMB‚ 2975.83203125 proc.cpu.threads‚17ô§zÇŸÉ‚(:” çóõÒßø‹Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚178.731' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚59.577000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wé°2Ê- _wandbruntime‚4130  _runtime‚4130Ñ!¹ÔÅË¿ & lr-AdamW/pg1‚0.0005474890005778474 & lr-AdamW/pg2‚0.0005474890005778474 trainer/global_step‚4599 # _timestamp‚1.78451095653175e+09 _runtime‚4130.108984003 _step‚375T£ ¬ÊÌÄ & lr-AdamW/pg2‚0.0005474890005778474 trainer/global_step‚4599 # _timestamp‚1.78451095653175e+09 _runtime‚4130.108984003 _step‚375 & lr-AdamW/pg1‚0.0005474890005778474÷¬(kª2Í- _wandbruntime‚4130  _runtime‚4130bÁ£Žç'ÎáO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5336593223680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚4130.109599074 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚703464067541504 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158373183490560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚708800660765184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37202591 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5367345863680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82684258 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚699350449596240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚699350432556880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37202593 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19035587 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19035301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8027289 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45491810 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5328537381833 trainer/global_step‚4599 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8027292 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153005837626880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚704678969938713 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5328877943107 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚703464084580864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚703464084580864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚708801019228672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5336934647808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚704679327539347 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37202591 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5336593223680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚708801019228672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153005856239104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45481667 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344–InAì'ÏæO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5336593223680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚4130.109599074 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚703464067541504 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158373183490560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚708800660765184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37202591 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5367345863680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82684258 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚699350449596240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚699350432556880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37202593 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19035587 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19035301 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8027289 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45491810 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5328537381833 trainer/global_step‚4599 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8027292 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153005837626880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚704678969938713 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5328877943107 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚703464084580864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚703464084580864 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚708801019228672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5336934647808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚704679327539347 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37202591 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5336593223680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚708801019228672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153005856239104 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45481667 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2øS˜L2Ð- _wandbruntime‚4130  _runtime‚4130è—s†ÑëM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚703612359083008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5337703322624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5329645770085 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37210384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚703612342043648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5368569448448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37210384 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45501480 XDDeviceStatsMonitorºÊà–k .on_train_batch_end/inactive_split_bytes.all.freed‚158407477889024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11010171 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5338044753920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19039496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158407605957632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5368459992064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37210386 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚708950403836928 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚704827866595183 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚703612342043648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19039194 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845109571468916e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚708950403836928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 trainer/global_step‚4599 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8029026 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚708950045366272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5337703322624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82701707 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚704827508994437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚699497880263712 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚699497863224352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5329986331471 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82701707 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82711866 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚4130.722615129 _step‚377 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153039036509184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚703612359083008 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ¡ö&ÒðM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚704827866595183 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚703612342043648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19039194 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845109571468916e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚708950403836928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 trainer/global_step‚4599 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8029026 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚708950045366272 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5337703322624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82701707 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚704827508994437 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚699497880263712 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚699497863224352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5329986331471 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82701707 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82711866 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚4130.722615129 _step‚377 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153039036509184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚703612359083008 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚703612359083008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5337703322624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5329645770085 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37210384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚703612342043648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5368569448448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37210384 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45501480 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158407477889024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11010171 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5338044753920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19039496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158407605957632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5368459992064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37210386 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚708950403836928 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336ù'w2Ó- _wandbruntime‚4130  _runtime‚4130äO2 Ô„ /loss/FRt1/detection_loss‚0.5396093381770642 /loss/FRt2/detection_loss‚1.5380846975204774 _step‚378 2loss/SRqqt2/assignment_loss‚0.5547949467863221 2loss/FBt1/assignment_loss‚0.003991427239366574 1loss/SRqqt2/detection_loss‚0.6750118185227847 % _timestamp‚1.7845109571481357e+09 0loss/FRt2/assignment_loss‚0.7608729472770783 0loss/SRqqt1/detection_loss‚0.662691711067243 /loss/FBt2/detection_loss‚0.8950024656473826 epoch‚11 trainer/global_step‚4599 _runtime‚4130.723396191 0loss/FRt1/assignment_loss‚1.7389448366836826 3loss/FBt2/assignment_loss‚0.0008582344953062751 /loss/FBt1/detection_loss‚0.5523299766100536 %loss/total_loss‚8.461879215366174 2loss/SRqqt1/assignment_loss‚0.5396868153394119EÅ6âÕ‰ 2loss/SRqqt2/assignment_loss‚0.5547949467863221 2loss/FBt1/assignment_loss‚0.003991427239366574 1loss/SRqqt2/detection_loss‚0.6750118185227847 % _timestamp‚1.7845109571481357e+09 0loss/FRt2/assignment_loss‚0.7608729472770783 0loss/SRqqt1/detection_loss‚0.662691711067243 /loss/FBt2/detection_loss‚0.8950024656473826 epoch‚11 trainer/global_step‚4599 _runtime‚4130.723396191 0loss/FRt1/assignment_loss‚1.7389448366836826 3loss/FBt2/assignment_loss‚0.0008582344953062751 /loss/FBt1/detection_loss‚0.5523299766100536 %loss/total_loss‚8.461879215366174 2loss/SRqqt1/assignment_loss‚0.5396868153394119 /loss/FRt1/detection_loss‚0.5396093381770642 /loss/FRt2/detection_loss‚1.5380846975204774 _step‚378ú¦ß \Ö‚(:Ñ öóõÒ–äÊË proc.cpu.threads‚17) disk./.usagePercent‚76.00461670677237 disk.nvme1n1.in‚0% disk.nvme1n1.out‚11836.7294921875 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.dm-0.out‚30290.25390625! disk.dm-1.out‚6177.7041015625 disk.rbd0.out‚0' disk.nvme1c1n1.out‚11836.7294921875& proc.memory.availableMB‚ 482.734375 cpu‚25.65151249205581 network.recv‚ 1.091815e+06% disk./.usageGB‚1166.8608779907227 disk.md0.out‚ 131.82421875& disk.nvme0c0n1.out‚24912.373046875 disk.nvme0n1.in‚ 0.00390625$ disk.nvme0n1.out‚24912.373046875$ memory_percent‚97.05362319946289* proc.memory.percent‚18.228673934936523 network.sent‚ 7.157141e+06 disk.dm-0.in‚ 0.00390625" proc.memory.rssMB‚ 2986.5859375ØÇÙž“ׂ(:ˆ öóõÒ´ÃÌ gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚263.613' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚87.871 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mv6¢cØ‚(:Ø …ôõÒè×ÖË network.sent‚ 7.221353e+06$ disk./.usageGB‚1166.861026763916 disk.md0.out‚ 132.4296875 disk.rbd0.out‚0 disk.nvme1n1.in‚0$ memory_percent‚97.03786373138428) proc.memory.percent‚18.30878257751465 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚30291.1220703125" disk.dm-1.out‚6177.87548828125 disk.rbd0.in‚0& disk.nvme0n1.out‚24913.80224609375 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚11837.6181640625" proc.memory.rssMB‚ 2999.7109375 network.recv‚ 1.096979e+06) disk./.usagePercent‚76.00462639725882 disk.nvme0c0n1.in‚ 0.00390625( proc.memory.availableMB‚ 485.31640625 cpu‚25.650906494948803 disk.md0.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚24913.80224609375 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚11837.6181640625 proc.cpu.threads‚17ßž$ÁžÙ‚(:“ …ôõÒ§¬ŸÌ gpu.0.gpu‚41.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚280.465' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.48833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·… [Ú‚(:Ð ”ôõÒê¹»Ë cpu‚25.650189315748353" disk.dm-1.out‚6177.91845703125 disk.nvme0c0n1.in‚ 0.00390625 network.sent‚ 7.226165e+06( disk./.usagePercent‚75.8276090387451$ disk./.usageGB‚1164.143367767334 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚30293.89013671875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625$ disk.nvme0n1.out‚24916.693359375' disk.nvme1c1n1.out‚11838.5576171875 disk.nvme1n1.in‚0# memory_percent‚96.8897819519043" proc.memory.rssMB‚ 3007.8515625 network.recv‚ 1.100209e+06 disk.md0.out‚ 132.90625 disk.nvme1c1n1.in‚0& proc.memory.availableMB‚ 509.578125* proc.memory.percent‚18.358469009399414 proc.cpu.threads‚17 disk.md0.in‚0& disk.nvme0c0n1.out‚24916.693359375% disk.nvme1n1.out‚11838.5576171875Y(¬žÛ‚(:“ ”ôõÒâëüË gpu.0.gpu‚62.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚285.911' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.30366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý Ùï2Ü- _wandbruntime‚4172  _runtime‚4172¹•›ÅÝ¿ % _timestamp‚1.7845109991451519e+09 _runtime‚4172.721247921 _step‚379 % lr-AdamW/pg1‚0.000545946365789506 % lr-AdamW/pg2‚0.000545946365789506 trainer/global_step‚4649XU…ÄÊÞÄ % lr-AdamW/pg1‚0.000545946365789506 % lr-AdamW/pg2‚0.000545946365789506 trainer/global_step‚4649 % _timestamp‚1.7845109991451519e+09 _runtime‚4172.721247921 _step‚379û©z@¥2ß- _wandbruntime‚4172  _runtime‚4172pw[K!àáO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83558560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5423279414272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845109991492178e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45966319 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚710877809688064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5392162325504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37592243 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚154664869743104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45966319 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚716270313437696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5384020595241 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8113992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚716269954974208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚712106344126355 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83568705 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚380 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11122926 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚706721982969840 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45976462 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45976462 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚4649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_trainíà5ß_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37592241 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19236619 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚706721965930480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚716269954974208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚710877809688064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚4172.724792467 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Išì'áæO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚710877792648704 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚710877792648704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11122630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5384361156515 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5392162325504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19236918 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8113989 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚380 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11122926 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚706721982969840 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45976462 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45976462 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚4649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37592241 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19236619 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚706721965930480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚716269954974208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚710877809688064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚4172.724792467 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83558560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5423279414272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845109991492178e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45966319 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚710877809688064 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5392162325504 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37592243 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚154664869743104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45966319 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚716270313437696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5384020595241 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8113992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚716269954974208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚712106344126355 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83568705 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37600034 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5424291090944 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19240670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _step‚381 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37600036 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚711026084190208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5393613855744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45975975 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚711026067150848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚716419698045952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5385128983493 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45986132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45975975 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚154698050013184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83576009 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45986132 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83586168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚712254883182191 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚160122322491904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11125222 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8115726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚4173.354924445 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5424394255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37600036 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83586168 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157TztWõ&äïM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚711026067150848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚716419698045952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5385128983493 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45986132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45975975 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚154698050013184 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83576009 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45986132 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83586168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚712254883182191 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚160122322491904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11125222 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8115726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚4173.354924445 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5424394255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37600036 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83586168 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37600034 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5424291090944 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19240670 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _step‚381 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37600036 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚711026084190208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5393613855744 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45975975ýÆ&·¡2å- _wandbruntime‚4173  _runtime‚4173øǻæ† 0loss/FRt2/assignment_loss‚0.8254093616097061 3loss/FBt2/assignment_loss‚0.0008132100346451631 1loss/SRqqt1/detection_loss‚0.6774548075322844 0loss/FBt2/detection_loss‚0.48582920372220023 _step‚382 0loss/FRt1/assignment_loss‚1.7340290037790935 3loss/SRqqt1/assignment_loss‚0.48782366037036057 /loss/FRt2/detection_loss‚1.5632720978539667 /loss/FBt1/detection_loss‚0.4944325976751718 1loss/FBt1/assignment_loss‚0.00989894373271209 0loss/FRt1/detection_loss‚0.48700400281789674 $loss/total_loss‚8.07284876590861 trainer/global_step‚4649 _runtime‚4173.355677027 2loss/SRqqt2/assignment_loss‚0.5557968296329358 1loss/SRqqt2/detecti!˜&»Qon_loss‚0.7510850471476377 epoch‚11 % _timestamp‚1.7845109997802346e+09Fý:ç‹ 0loss/FBt2/detection_loss‚0.48582920372220023 _step‚382 0loss/FRt1/assignment_loss‚1.7340290037790935 3loss/SRqqt1/assignment_loss‚0.48782366037036057 /loss/FRt2/detection_loss‚1.5632720978539667 /loss/FBt1/detection_loss‚0.4944325976751718 1loss/FBt1/assignment_loss‚0.00989894373271209 0loss/FRt1/detection_loss‚0.48700400281789674 $loss/total_loss‚8.07284876590861 trainer/global_step‚4649 _runtime‚4173.355677027 2loss/SRqqt2/assignment_loss‚0.5557968296329358 1loss/SRqqt2/detection_loss‚0.7510850471476377 epoch‚11 % _timestamp‚1.7845109997802346e+09 0loss/FRt2/assignment_loss‚0.8254093616097061 3loss/FBt2/assignment_loss‚0.0008132100346451631 1loss/SRqqt1/detection_loss‚0.6774548075322844þ\ôfè‚(:Û £ôõÒö¹ßË disk.dm-0.in‚ 0.00390625! disk.dm-1.out‚6178.0791015625 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚24917.9794921875 cpu‚25.65048257998449 proc.cpu.threads‚17) disk./.usagePercent‚75.81215271284324% disk./.usageGB‚1163.9060745239258 disk.md0.in‚0# disk.dm-0.out‚30294.79638671875 disk.rbd0.out‚0$ memory_percent‚96.83096408843994( proc.memory.availableMB‚ 519.21484375# proc.memory.rssMB‚ 3014.10546875 network.sent‚ 7.241801e+06 network.recv‚ 1.103155e+06 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚11839.32470703125 disk.md0.out‚ 133.35546875 disk.rbd0.in‚0* proc.memory.percent‚18.396639823913574& disk.nvme1n1.out‚11839.32470703125 disk.dm-1.in‚0% disk.nvme0n1.out‚24917.9794921875 disk.nvme1n1.in‚0þ&Õé‚(:’ £ôõÒ滞Ì gpu.0.gpu‚60.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚125.81' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.93666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º¥Paê‚(:Ö ²ôõÒšÝÌË% disk./.usageGB‚1163.9060249328613 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚24924.2939453125% disk.nvme0n1.out‚24924.2939453125 disk.dm-1.in‚0! disk.dm-1.out‚6178.1376953125! proc.memory.rssMB‚ 3017.984375 network.sent‚ 7.29244e+06) disk./.usagePercent‚75.81214948268108 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚11840.24755859375 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625$ memory_percent‚96.76942825317383& disk.nvme1n1.out‚11840.24755859375 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.dm-0.out‚30300.61376953125 cpu‚25.649771183670886 proc.cpu.threads‚17 network.recv‚ 1.107939e+06 disk.md0.out‚ 134.00390625& proc.memory.availableMB‚ 529.296875) proc.memory.percent‚18.42031478881836DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4699 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚723739249183232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚723739607646720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8200692 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11237219 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161802853773824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚719533360713363 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84443007 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19437626 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161802973460992 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46461114 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚714093516343440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚714093499304080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8200689 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚718291517755904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚718291534795264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5448072851456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚723739249183232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37981891 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145JXîë'ñåO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46450971 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚156323864634880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5439844369923 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚718291534795264 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚718291517755904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5448072851456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚156323883247104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37981893 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19437911 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 $ _timestamp‚1.784511041238229e+09 _runtime‚4214.815098442 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5447731427328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5479090213888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37981891 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46461114 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚714093516343440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚714093499304080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8200689 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚718291517755904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚718291534795264 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5448072851456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚723739249183232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37981891 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5478989138944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37981893 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4699 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚723739249183232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚723739607646720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8200692 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11237219 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161802853773824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚719533360713363 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84443007 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19437626 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161802973460992 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161837275804672 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84460470 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134360064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8202423 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84450311 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚723888992254976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚718439809297408 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84450311 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19441708 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161837141444608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 $ _timestamp‚1.784511041838109e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37989684 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚719681899769199 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37989684 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚714240929971552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5480212287488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46470784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚718439809297408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 _runtime‚4215.413815899 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5440952758287 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚718439792258048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚719681542168453 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5448841526272 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚714240947010912 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ÅoI¹ REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5440952758287 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚718439792258048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚719681542168453 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5448841526272 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚714240947010912 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161837275804672 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84460470 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134360064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8202423 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84450311 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚723888992254976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚718439809297408 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84450311 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19441708 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161837141444608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0e+Ð 2÷- _wandbruntime‚4215  _runtime‚4215÷AÔ ø‚ epoch‚11 /loss/FRt2/assignment_loss‚0.824749683952767 2loss/SRqqt1/assignment_loss‚0.5953354035559824 /loss/FBt1/detection_loss‚0.4234643370657758 0loss/FBt2/detection_loss‚0.39032009601942264 trainer/global_step‚4699 .loss/FRt2/detection_loss‚1.520070160179905 $loss/total_loss‚8.08722522882336 _runtime‚4215.422911503 2loss/SRqqt2/assignment_loss‚0.5243215733267507 2loss/FBt1/assignment_loss‚0.001538112819336931 2loss/FBt2/assignment_loss‚0.041109767846058944 1loss/SRqqt2/detection_loss‚0.7714663944675173 % _timestamp‚1.7845110418475196e+09 _step‚386 0loss/FRt1/assignment_loss‚1.7955366386897937 /loss/FRt1/detection_loss‚0.4874192260639682 1loss/SRqqt1/detection_loss‚0.7118938348360825“¯. ù‡ 2loss/SRqqt2/assignment_loss‚0.5243215733267507 2loss/FBt1/assignment_loss‚0.001538112819336931 2loss/FBt2/assignment_loss‚0.041109767846058944 1loss/SRqqt2/detection_loss‚0.7714663944675173 % _timestamp‚1.7845110418475196e+09 _step‚386 0loss/FRt1/assignment_loss‚1.7955366386897937 /loss/FRt1/detection_loss‚0.4874192260639682 1loss/SRqqt1/detection_loss‚0.7118938348360825 epoch‚11 /loss/FRt2/assignment_loss‚0.824749683952767 2loss/SRqqt1/assignment_loss‚0.5953354035559824 /loss/FBt1/detection_loss‚0.4234643370657758 0loss/FBt2/detection_loss‚0.39032009601942264 trainer/global_step‚4699 .loss/FRt2/detection_loss‚1.520070160179905 $loss/total_loss‚8.08722522882336 _runtime‚4215.422911503‚8¹…ö`ú‚(:Õ ÐôõÒ¥’ÙË disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0$ memory_percent‚96.76961898803711! proc.memory.rssMB‚ 3017.984375 network.recv‚ 1.116305e+06 disk.md0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625& proc.memory.availableMB‚ 529.265625) proc.memory.percent‚18.42031478881836) disk./.usagePercent‚75.81212065969571 disk.nvme1n1.in‚0" disk.dm-1.out‚6178.32373046875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚24939.50537109375& disk.nvme1c1n1.out‚11843.669921875 cpu‚25.648240801311808 proc.cpu.threads‚17% disk./.usageGB‚1163.9055824279785$ disk.nvme1n1.out‚11843.669921875 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚30317.8076171875& disk.nvme0n1.out‚24939.50537109375 network.sent‚ 7.359869e+06 disk.md0.out‚ 134.57421875¸6®žû‚(:“ ÐôõÒ‹ú”Ì gpu.0.gpu‚59.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚285.615' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.20500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ÕÓ|`ü‚(:Õ ßôõÒù—Ë network.sent‚ 7.399109e+06) disk./.usagePercent‚76.01167237481044' disk.nvme1c1n1.out‚13757.6298828125 disk.md0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚6178.41650390625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 network.recv‚ 1.121571e+06 disk.nvme1n1.in‚0% disk.nvme1n1.out‚13757.6298828125 disk.dm-0.out‚ 32268.3984375( disk.nvme0c0n1.out‚24977.22119140625 disk.nvme0n1.in‚ 0.00390625$ memory_percent‚96.76969051361084( proc.memory.availableMB‚ 529.25390625 disk.md0.out‚ 135.0390625 disk.dm-0.in‚ 0.00390625) proc.memory.percent‚18.42031478881836 cpu‚25.647490816101428 proc.cpu.threads‚17% disk./.usageGB‚1166.9692001342773 disk.nvme1c1n1.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚24977.22119140625! proc.memory.rssMB‚ 3017.984375´%Etœý‚(:‘ ßôõÒðåúË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.025' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.34166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§´Zd2þ- _wandbruntime‚4256  _runtime‚4256TqSÇÿÁ & lr-AdamW/pg1‚0.0005428152021943554 & lr-AdamW/pg2‚0.0005428152021943554 trainer/global_step‚4749 % _timestamp‚1.7845110831810126e+09 _runtime‚4256.756603348 _step‚387‡X‡{Ì€Æ trainer/global_step‚4749 % _timestamp‚1.7845110831810126e+09 _runtime‚4256.756603348 _step‚387 & lr-AdamW/pg1‚0.0005428152021943554 & lr-AdamW/pg2‚0.0005428152021943554ƒ×4Øä2- _wandbruntime‚4256  _runtime‚4256}avqç'‚áO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11351884 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157982896751104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚731208901855744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5503641953280 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38371541 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚726960019699737 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚388 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚4256.757439039 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚731208901855744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚725705242863104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163517856435200 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚721465032677680 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85317309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚721465049717040 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚726960377300371 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚725705259902464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚731208543392256 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85307164 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚725705259902464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38371541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚4749 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46935623 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8287392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157982878138880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46935623 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5495327583331 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19638986 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5534852317696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731208543392256 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚725705242863104 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845110831819463e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19639276 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38371543 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11351597 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5494987022057 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38371543 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5503300529152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0îß:ʃæO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157982896751104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚731208901855744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5503641953280 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38371541 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚726960019699737 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚388 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚4256.757439039 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚731208901855744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚725705242863104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163517856435200 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚721465032677680 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85317309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚721465049717040 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039’oQ"360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚726960377300371 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚725705259902464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚731208543392256 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85307164 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚725705259902464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38371541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚4749 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46935623 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8287392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157982878138880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46935623 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5495327583331 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19638986 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5534852317696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚731208543392256 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚725705242863104 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845110831819463e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19639276 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38371543 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11351597 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5494987022057 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38371543 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5503300529152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11351884„›¦)2„- _wandbruntime‚4257  _runtime‚4257:±!ûñ&…ëM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚725853517365248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚725853517365248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚4749 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5504752059392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163552153431552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _step‚389 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚725853534404608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38379334 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8289123 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38379336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85324613 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38379336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19643249 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5496435971695 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5504410628096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46955436 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11353827 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46955436 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚731358286464000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163552023265792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5536076410368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158016077021184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚727108916356207 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5496095410309 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5504752059392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85334772 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5535964856832 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚721612463345152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚4257.369830352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38379334 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚721612480384512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19642950 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845110837940795e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85334772 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46945279 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731358286464000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5504410628096/>pªö&†ðM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19642950 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845110837940795e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85334772 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46945279 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚731358286464000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5504410628096 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚725853517365248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚725853517365248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚4749 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5504752059392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163552153431552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _step‚389 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚725853534404608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38379334 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8289123 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38379336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85324613 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38379336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19643249 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5496435971695 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5504410628096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46955436 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11353827 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46955436 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚731358286464000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚163552023265792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5536076410368 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158016077021184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚727108916356207 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5496095410309 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5504752059392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85334772 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5535964856832 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚721612463345152 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚4257.369830352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38379334 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚721612480384512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272…ïÑŠc2‡- _wandbruntime‚4257  _runtime‚4257Ðk .loss/FRt2/detection_loss‚1.617369540901493 %loss/total_loss‚8.206971830318977 _runtime‚4257.372800837 _step‚390 0loss/FRt1/assignment_loss‚1.7045625993665658 2loss/SRqqt2/assignment_loss‚0.5691363308936532 2loss/FBt1/assignment_loss‚0.015382844022481156 1loss/SRqqt1/detection_loss‚0.6680053251152527 /loss/FBt1/detection_loss‚0.5227603126078659 /loss/FBt2/detection_loss‚0.6897679525995399 epoch‚11 1loss/SRqqt1/assignment_loss‚0.523279480778241 2loss/FBt2/assignment_loss‚0.022173253787326774 .loss/FRt1/detection_loss‚0.473178923717466 trainer/global_step‚4749 % _timestamp‚1.7845110837974718e+09 0loss/FRt2/assignment_loss‚0.8074323498797649 1loss/SRqqt2/detection_loss‚0.5939229166493277Ç 5Ô ‰† 1loss/SRqqt1/assignment_loss‚0.523279480778241 2loss/FBt2/assignment_loss‚0.022173253787326774 .loss/FRt1/detection_loss‚0.473178923717466 trainer/global_step‚4749 % _timestamp‚1.7845110837974718e+09 0loss/FRt2/assignment_loss‚0.8074323498797649 1loss/SRqqt2/detection_loss‚0.5939229166493277 .loss/FRt2/detection_loss‚1.617369540901493 %loss/total_loss‚8.206971830318977 _runtime‚4257.372800837 _step‚390 0loss/FRt1/assignment_loss‚1.7045625993665658 2loss/SRqqt2/assignment_loss‚0.5691363308936532 2loss/FBt1/assignment_loss‚0.015382844022481156 1loss/SRqqt1/detection_loss‚0.6680053251152527 /loss/FBt1/detection_loss‚0.5227603126078659 /loss/FBt2/detection_loss‚0.6897679525995399 epoch‚11†¢¬ª¾^Š‚(:Ó îôõÒ˜ÏË disk.nvme1n1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1166.9059143066406 disk.dm-1.in‚0" disk.dm-1.out‚6178.52978515625 disk.nvme1c1n1.in‚0 disk.md0.in‚0$ memory_percent‚96.77128791809082' proc.memory.availableMB‚ 528.9921875! proc.memory.rssMB‚ 3017.984375 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.dm-0.in‚ 0.00390625) proc.memory.percent‚18.42031478881836 cpu‚25.64661717482713 network.recv‚ 1.125043e+06) disk./.usagePercent‚76.00755019095527 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1n1.out‚13825.9208984375 disk.md0.out‚ 135.53125! disk.dm-0.out‚32481.642578125 network.sent‚ 7.417843e+06( disk.nvme0c0n1.out‚25123.30712890625& disk.nvme0n1.out‚25123.30712890625' disk.nvme1c1n1.out‚13825.9208984375&á–}ž‹‚(:“ îôõÒ­†™Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.633' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.87766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹H×]Œ‚(:Ò ýôõÒœ¶ÛË network.sent‚ 7.468135e+06 network.recv‚ 1.130201e+06 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚14090.982421875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚14090.982421875) disk./.usagePercent‚75.82753275722348 disk.dm-1.in‚0! disk.dm-1.out‚6178.6572265625' disk.nvme0c0n1.out‚25725.8271484375( proc.memory.availableMB‚ 528.98046875! proc.memory.rssMB‚ 3017.984375 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme0n1.out‚25725.8271484375) proc.memory.percent‚18.42031478881836 cpu‚25.64587043509717% disk./.usageGB‚1164.1421966552734 disk.md0.in‚0 disk.md0.out‚ 135.88671875! disk.dm-0.out‚33348.330078125$ memory_percent‚96.77135944366455®÷Wž‚(:“ ýôõÒìÉœÌ gpu.0.gpu‚73.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚273.701' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.23366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òÓŸ\dŽ‚(:Ù ŒõõÒƒÂ²Ë disk.rbd0.out‚0( proc.memory.availableMB‚ 528.97265625! proc.memory.rssMB‚ 3017.984375 network.sent‚ 7.471099e+06% disk./.usageGB‚1164.1419563293457( disk.nvme1c1n1.out‚14093.06396484375 disk.nvme1n1.in‚0 disk.md0.out‚ 136.34765625 proc.cpu.threads‚17 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚33364.52001953125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚25740.9814453125$ memory_percent‚96.77140712738037) proc.memory.percent‚18.42031478881836) disk./.usagePercent‚75.82751710336073% disk.nvme0n1.out‚25740.9814453125 disk.nvme1c1n1.in‚0 cpu‚25.645076038925378 network.recv‚ 1.133608e+06 disk.nvme0n1.in‚ 0.00390625& disk.nvme1n1.out‚14093.06396484375! disk.dm-1.out‚6178.7041015625 disk.rbd0.in‚0Îwä.‚(:’ ŒõõÒÁ¬†Ì gpu.0.gpu‚16.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚289.616' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.53866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èè¡„2- _wandbruntime‚4299  _runtime‚4299ïìÇ‘Á _step‚391 & lr-AdamW/pg1‚0.0005412268744963332 & lr-AdamW/pg2‚0.0005412268744963332 trainer/global_step‚4799 % _timestamp‚1.7845111256667776e+09 _runtime‚4299.243387675ã~n½Ì’Æ & lr-AdamW/pg1‚0.0005412268744963332 & lr-AdamW/pg2‚0.0005412268744963332 trainer/global_step‚4799 % _timestamp‚1.7845111256667776e+09 _runtime‚4299.243387675 _step‚391‡¼Ÿ8Î2“- _wandbruntime‚4299  _runtime‚4299< Q1”áO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 % _timestamp‚1.7845111256673589e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚733118967970304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38761193 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5550470235465 D:DeviceStatsMonitor.on_Û)"Ô"train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚738677837601280 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11465874 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5590680051712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86191611 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38761193 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚306 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5590783223808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚734387036286745 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38761191 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚728836583090640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚4799 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5559211055104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5550810796739 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚738678196064768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47420275 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5559211055104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38761191 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚303 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47430418 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47420275 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86181466 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚738678196064768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159641891642880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚728836566051280 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚4299.243447295 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47430418 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165232571694592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 % _timestamp‚1.7845111256673589e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚733118967970304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38761193 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5550470235465 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚738677837601280 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11465874 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5590680051712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86191611 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38761193 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚306 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5590783223808 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚734387036286745 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38761191 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚728836583090640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚4799 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5559211055104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5550810796739 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚738678196064768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47420275 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5559211055104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38761191 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚303 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47430418 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47420275 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86181466 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚738678196064768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159641891642880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚728836566051280 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚4299.243447295 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47430418 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165232571694592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Iñ&—ëM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚4799 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47440088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86198915 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚738827580673024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19843620 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11468074 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11467797 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚738827222202368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159675071912960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 _step‚393 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47429931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚728984013758112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚733267259511808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47440088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5551578623717 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚733267242472448 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86209074 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚728983996718752 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8375826 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165266861308928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159675090525184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5560321161216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚734535932943215 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5559979729920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38768986 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5591789395968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚733267242472448 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38768984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚733267259511808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚734535575342469 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47429931 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8375823 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86209074 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7845111262785795e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5591896755200 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746t„§›ú ˜ðM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5551919185103 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47440088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5551578623717 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚733267242472448 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86209074 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚728983996718752 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8375826 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165266861308928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159675090525184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5560321161216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚734535932943215 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_$0übatch_end/segment.large_pool.peak‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5559979729920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38768986 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5591789395968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚733267242472448 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38768984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚733267259511808 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚734535575342469 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47429931 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8375823 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86209074 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7845111262785795e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5591896755200 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚4799 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47440088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86198915 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚738827580673024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19843620 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11468074 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11467797 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚738827222202368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159675071912960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 _step‚393 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47429931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚728984013758112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚733267259511808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 F&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17250001430511475 +jet/accuracy_3_of_4‚0.09818699210882187 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7135814428329468 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05645567923784256 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22299210727214813 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3134959042072296 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9900336265563965 +jet/accuracy_3_of_3‚0.07809082418680191 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7019131779670715 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2163742631673813 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7222573161125183 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992892742156982 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03721561282873154 particle/accuracy_2_of_2‚1 +particle/specificity‚0.9082362055778503 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.576447606086731 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993686854839325 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5732473731040955 *jet/accuracy_1_of_2‚0.7944934368133545 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.911474883556366 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20074887573719025 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7166365385055542 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4666813313961029 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964554905891418 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6401156187057495 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.612074077129364 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7477200031280518 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35173261165618896 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6020217537879944 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0415479876101017 particle/accuracy_4_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957677125930786 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992892742156982 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22531239688396454 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18090012669563293 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6866618394851685 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3192930221557617 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9899159669876099 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.021513525396585464 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.235690638422966 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8885191082954407 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6881488561630249 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05645567923784256 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_4‚0.7954493165016174 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5857802033424377 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8795918226242065 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.197586789727211 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6667221188545227 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4652777910232544 +jet/accuracy_2_of_2‚0.14840878546237946 *jet/accuracy_1_of_4‚0.9592747688293457 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2665351927280426 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5808055400848389 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8190245628356934 *jet/accuracy_1_of_3‚0.9285954236984253 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6388462781906128 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *particle/sensitivity‚0.634925901889801 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6646130084991455 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7016243934631348 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8045352101325989 particle/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996904730796814 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962763786315918 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_2_of_5‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.25435778498649597 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6127796769142151 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8495435118675232 )loss/val_total_loss‚7.867735528350843 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04809531569480896 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.32537612318992615 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17169921100139618 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5228790640830994 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7577260732650757 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.25418832898139954 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_3_of_4‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984360933303833 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4706653356552124 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6709253787994385 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3563462197780609 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6643458008766174 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22259928286075592 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2865165174007416 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8703703284263611 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8190245628356934 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26736778020858765 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8664105534553528 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0432431660592556 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 jet/accuracy_5_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5989519953727722 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2665858566761017 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6560074687004089 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912253022193909 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19529104232788086 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9229111671447754 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06015828251838684 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1893738955259323 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8703703284263611 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19895900785923004 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.031209643930196762 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6901042461395264 epoch‚11 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2449691891670227 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6703850626945496 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27540239691734314 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 % _timestamp‚1.7845111422150314e+09 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9506555199623108 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1861661672592163 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 &particle/f_score‚0.644425094127655 _step‚395 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04846502095460892 7"Purity/0FRt0SRqqt2FBt/event_purity‚0.98065584897995 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5699999332427979 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_3‚0.5516411662101746 7validation_average_jet_accuracy‚0.36935365200042725 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4114583432674408 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 trainer/global_step‚4799 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6688324809074402 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4608178734779358 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9955742359161377 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6392602324485779 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6020217537879944 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7417003512382507 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20509710907936096 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024532970041036606 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7191041707992554 _runtime‚4315.791231265 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2542549967765808 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.042733341455459595 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6393272280693054 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4358096420764923 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8816893696784973 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7624298334121704 particle/accuracy_1_of_4‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8718064427375793 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9903279542922974 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.25418832898139954 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8816893696784973 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7526670694351196 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7641164660453796 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6009573340415955 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.324022114276886 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7166385054588318 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6144835352897644 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977300763130188 +jet/accuracy_4_of_4‚0.01670061983168125 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941260814666748 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44845345616340637 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23743674159049988 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16706296801567078 .particle/accuracy_4_of_4‚0.960080087184906 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6576611995697021 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5910078287124634Cæ-'ü Ñz 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26736778020858765 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8664105534553528 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0432431660592556 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 jet/accuracy_5_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5989519953727722 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2665858566761017 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6560074687004089 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912253022193909 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19529104232788086 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9229111671447754 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06015828251838684 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1893738955259323 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8703703284263611 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19895900785923004 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.031209643930196762 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6901042461395264 epoch‚11 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2449691891670227 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6703850626945496 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27540239691734314 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 % _timestamp‚1.7845111422150314e+09 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9506555199623108 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1861661672592163 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 &particle/f_score‚0.644425094127655 _step‚395 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04846502095460892 7"Purity/0FRt0SRqqt2FBt/event_purity‚0.98065584897995 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5699999332427979 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_3‚0.5516411662101746 7validation_average_jet_accuracy‚0.36935365200042725 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4114583432674408 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 trainer/global_step‚4799 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6688324809074402 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4608178734779358 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9955742359161377 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6392602324485779 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6020217537879944 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7417003512382507 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20509710907936096 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024532970041036606 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7191041707992554 _runtime‚4315.791231265 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2542549967765808 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.042733341455459595 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6393272280693054 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4358096420764923 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8816893696784973 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7624298334121704 particle/accuracy_1_of_4‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8718064427375793 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9903279542922974 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.25418832898139954 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8816893696784973 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7526670694351196 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7641164660453796 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6009573340415955 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.324022114276886 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7166385054588318 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6144835352897644 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977300763130188 +jet/accuracy_4_of_4‚0.01670061983168125 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941260814666748 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44845345616340637 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23743674159049988 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16706296801567078 .particle/accuracy_4_of_4‚0.960080087184906 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6576611995697021 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5910078287124634 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8634164929389954 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17250001430511475 +jet/accuracy_3_of_4‚0.09818699210882187 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7135814428329468 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05645567923784256 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22299210727214813 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3134959042072296 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9900336265563965 +jet/accuracy_3_of_3‚0.07809082418680191 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7019131779670715 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt1FBtÈ#qé[/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2163742631673813 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7222573161125183 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992892742156982 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03721561282873154 particle/accuracy_2_of_2‚1 +particle/specificity‚0.9082362055778503 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.576447606086731 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.993686854839325 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5732473731040955 *jet/accuracy_1_of_2‚0.7944934368133545 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.911474883556366 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20074887573719025 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7166365385055542 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4666813313961029 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9964554905891418 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6401156187057495 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.612074077129364 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7477200031280518 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35173261165618896 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6020217537879944 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0415479876101017 particle/accuracy_4_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957677125930786 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992892742156982 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22531239688396454 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18090012669563293 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6866618394851685 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3192930221557617 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9899159669876099 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.021513525396585464 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.235690638422966 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8885191082954407 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6881488561630249 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05645567923784256 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_4‚0.7954493165016174 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5857802033424377 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8795918226242065 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.197586789727211 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6667221188545227 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4652777910232544 +jet/accuracy_2_of_2‚0.14840878546237946 *jet/accuracy_1_of_4‚0.9592747688293457 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2665351927280426 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5808055400848389 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8190245628356934 *jet/accuracy_1_of_3‚0.9285954236984253 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6388462781906128 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *particle/sensitivity‚0.634925901889801 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6646130084991455 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7016243934631348 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8045352101325989 particle/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996904730796814 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962763786315918 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_2_of_5‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.25435778498649597 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6127796769142151 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8495435118675232 )loss/val_total_loss‚7.867735528350843 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04809531569480896 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.32537612318992615 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17169921100139618 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5228790640830994 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7577260732650757 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.25418832898139954 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_3_of_4‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984360933303833 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4706653356552124 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6709253787994385 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3563462197780609 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6643458008766174 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22259928286075592 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2865165174007416 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8703703284263611 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8190245628356934‹zùr¸^¡‚(:Ó ªõõÒ‚Ú˜Ë& disk.nvme1n1.out‚14341.06689453125 disk.md0.out‚ 137.109375 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚33376.9970703125$ memory_percent‚95.48420906066895' proc.memory.availableMB‚ 739.8671875 disk.nvme1c1n1.in‚0 disk.rbd0.in‚0 disk.md0.in‚0) proc.memory.percent‚18.42031478881836 proc.cpu.threads‚17$ disk./.usageGB‚1164.141933441162 disk.rbd0.out‚0( disk.nvme1c1n1.out‚14341.06689453125! proc.memory.rssMB‚ 3017.984375 cpu‚25.65237699835825& disk.nvme0n1.out‚25752.68505859375 network.sent‚ 7.538388e+06 network.recv‚ 1.142709e+06) disk./.usagePercent‚75.82751561251666 disk.dm-1.in‚0 disk.dm-1.out‚ 6424.26171875 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚25752.68505859375 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 ûZž¢‚(:“ ªõõÒÏ Ì gpu.0.gpu‚64.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚288.445' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.14833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%£À©b£‚(:× ¹õõÒ½ªÕË# disk.dm-0.out‚35291.75830078125 disk.dm-1.in‚0 proc.cpu.threads‚17' disk.nvme1c1n1.out‚16375.0654296875 network.recv‚ 1.147639e+06 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚25754.28173828125 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚25754.28173828125 disk.md0.out‚ 137.4453125 disk.dm-0.in‚ 0.00390625 network.sent‚ 7.597238e+06) disk./.usagePercent‚76.00740383976233 disk.nvme1n1.in‚0 disk.md0.in‚0! proc.memory.rssMB‚ 3017.984375) proc.memory.percent‚18.42031478881836% disk.nvme1n1.out‚16375.0654296875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0! disk.dm-1.out‚6544.3623046875$ memory_percent‚95.48571109771729( proc.memory.availableMB‚ 739.62109375 cpu‚25.65150111820483% disk./.usageGB‚1166.9036674499512ÚÚÚ ‘¤‚(:† ¹õõÒ‡Ï–Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.429' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.143 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)“`¥‚(:Õ ÈõõÒ§žË network.sent‚ 7.602174e+06 network.recv‚ 1.151294e+06 disk.rbd0.in‚0& disk.nvme0n1.out‚25758.25439453125 disk.md0.out‚ 137.859375! disk.dm-1.out‚6669.7607421875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625& disk.nvme1n1.out‚16501.31591796875 disk.dm-0.out‚35295.68359375) proc.memory.percent‚18.42031478881836 disk.nvme1n1.in‚0) disk./.usagePercent‚76.00741651193694 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚25758.25439453125 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0% disk./.usageGB‚1166.9038619995117$ memory_percent‚95.48420906066895' proc.memory.availableMB‚ 739.8671875! proc.memory.rssMB‚ 3017.984375 cpu‚25.650814183764986( disk.nvme1c1n1.out‚16501.31591796875«¶3ž¦‚(:“ ÈõõÒóÖáË gpu.0.gpu‚91.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚279.497' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.16566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ns2§- _wandbruntime‚4358  _runtime‚4358.Ž‡XƨÀ & lr-AdamW/pg1‚0.0005396235185611769 & lr-AdamW/pg2‚0.0005396235185611769 trainer/global_step‚4849 $ _timestamp‚1.784511185187049e+09 _runtime‚4358.764282862 _step‚396µ&Ë©Å _step‚396 & lr-AdamW/pg1‚0.0005396235185611769 & lr-AdamW/pg2‚0.0005396235185611769 trainer/global_step‚4849 $ _timestamp‚1.784511185187049e+09 _runtime‚4358.764282862Œ‘ø 2ª- _wandbruntime‚4358  _runtime‚4358P⥠æ'«àO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚743755970331513 % _timestamp‚1.7845111851893704e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5626597859049 trainer/global_step‚4849 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166990766283776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚748109793524224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚738129355433104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39273651 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11618441 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5626257297775 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39273651 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47986426 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39273649 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5634756571136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39273649 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47996569 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166990879679488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚742475053992448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748109793524224 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87260075 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748110151987712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161323483809280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚397 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚742475036953088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5635097995264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5634756571136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚738129372472464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161323502421504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚742475036953088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87270220 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚743755612730879 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚748110151987712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47996569 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5635097995264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11618154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224Ü2®:‚'¬åO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚743755970331513 % _timestamp‚1.7845111851893704e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5626597859049 trainer/global_step‚4849 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166990766283776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚748109793524224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚738129355433104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39273651 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11618441 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5626257297775 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39273651 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47986426 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39273649 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5634756571136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39273649 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47996569 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166990879679488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚742475053992448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748109793524224 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87260075 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748110151987712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚161323483809280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚397 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚742475036953088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5635097995264 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5634756571136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚738129372472464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161323502421504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚742475036953088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87270220 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚743755612730879 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚748110151987712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47996569 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5635097995264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11618154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_>[¸ipool.current‚18612224 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143º22­- _wandbruntime‚4359  _runtime‚4359ªÕÏ~ï&®éM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚743904509387349 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87287683 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚742623328494592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47996082 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20087120 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8466639 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161356664079360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5635866670080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11620761 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5668404868608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161356682691584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47996082 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚748259536595968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 % _timestamp‚1.7845111858002963e+09 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48006239 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107097088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87287683 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39281444 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748259536595968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87277524 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5636208101376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5627365686027 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39281442 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚742623311455232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚738276786100576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5627706247413 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39281444 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚742623311455232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0“¢Cßô&¯îM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 % _timestamp‚1.7845111858002963e+09 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48006239 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107097088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87287683 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39281444 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748259536595968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87277524 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5636208101376 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5627365686027 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39281442 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚742623311455232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚738276786100576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5627706247413 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39281444 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚742623311455232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚743904509387349 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87287683 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚742623328494592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47996082 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20087120 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8466639 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161356664079360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5635866670080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11620761 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5668404868608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161356682691584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47996082 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚748259536595968 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88144522 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20285372 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5690325672960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚755579446196736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5690325672960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48471078 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚745500905846064 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168705845733376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39663299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚749888779099648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5690667097088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162982497313280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162982515925504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚749888779099648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39663299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48481221 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚755579446196736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚749888762060288 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5682081072457 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11733471 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8551605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5723329807872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitoržp%y.on_train_batch_start/segment.small_pool.allocated‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168705715560448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚4899 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚755579087733248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180˜Zœ(ë'¾åO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168705845733376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39663299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚749888779099648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5690667097088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162982497313280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162982515925504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚749888779099648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39663299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48481221 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚755579446196736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚749888762060288 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5682081072457 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11733471 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8551605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5723329807872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168705715560448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚4899 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚755579087733248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚755579087733248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5690667097088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48481221 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _runtime‚ 4400.98412412 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7845112274068313e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48471078 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚751182629317887 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚745500888806704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88144522 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20285372 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5690325672960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚755579446196736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5690325672960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48471078 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚745500905846064 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184‘…³´[2¿- _wandbruntime‚4401  _runtime‚4401•÷0ñ&ÀëM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5724337454592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚751331168373611 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚745648336513536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5683189460821 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚755728830804992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20289367 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168740015037952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5691777203200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845112280188997e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48490891 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚402 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88161985 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88161985 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚745648319474176 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚755728830804992 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20289086 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚751331525974357 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88151826 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39671094 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8553342 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚4899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5691435771904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168740138912256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5691777203200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚750037036562432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163015696195584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5691435771904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚750037053601792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11735747 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚750037053601792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39671092 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚4401.594619555 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚750037036562432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48480734 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8553339 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163015677583360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5724442716672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48490891 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11736025 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48480734 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0\~¬X½"ÁðM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11735747 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚750037053601792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39671092 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚4401.594619555 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚750037036562432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48480734 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8553339 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚163015677583360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5724442716672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48490891 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11736025 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48480734 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5724337454592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚751331168373611 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚745648336513536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5683189460821 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚755728830804992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20289367 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168740015037952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5691777203200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845112280188997e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48490891 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚402 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88161985 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88161985 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚745648319474176 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚755728830804992 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20289086 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚751331525974357 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88151826 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39671094 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8553342 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚4899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5691435771904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚168740138912256 R¦º»œ9DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5691777203200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚750037036562432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163015696195584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5691435771904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚750037053601792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157’ö^%X2Â- _wandbruntime‚4401  _runtime‚4401µ€? Ã… _step‚403 0loss/FRt1/assignment_loss‚1.7424724868856944 2loss/SRqqt2/assignment_loss‚0.5339117608350934 1loss/SRqqt1/detection_loss‚0.8175838594552881 % _timestamp‚1.7845112280199034e+09 2loss/SRqqt1/assignment_loss‚0.4401755935248821 /loss/FRt2/detection_loss‚1.6555151091680043 /loss/FBt1/detection_loss‚0.5094667837457282 .loss/FBt2/detection_loss‚0.544560233486647 0loss/FRt2/assignment_loss‚0.7751178142443225 3loss/FBt2/assignment_loss‚0.0032706745560278448 trainer/global_step‚4899 _runtime‚4401.595177786 epoch‚12 3loss/FBt1/assignment_loss‚0.0021982208238905286 /loss/FRt1/detection_loss‚0.4971207770802511 1loss/SRqqt2/detection_loss‚0.6589296415027232 %loss/total_loss‚8.180322955308553›¿¦³ÄŠ 2loss/SRqqt1/assignment_loss‚0.4401755935248821 /loss/FRt2/detection_loss‚1.6555151091680043 /loss/FBt1/detection_loss‚0.5094667837457282 .loss/FBt2/detection_loss‚0.544560233486647 0loss/FRt2/assignment_loss‚0.7751178142443225 3loss/FBt2/assignment_loss‚0.0032706745560278448 trainer/global_step‚4899 _runtime‚4401.595177786 epoch‚12 3loss/FBt1/assignment_loss‚0.0021982208238905286 /loss/FRt1/detection_loss‚0.4971207770802511 1loss/SRqqt2/detection_loss‚0.6589296415027232 %loss/total_loss‚8.180322955308553 _step‚403 0loss/FRt1/assignment_loss‚1.7424724868856944 2loss/SRqqt2/assignment_loss‚0.5339117608350934 1loss/SRqqt1/detection_loss‚0.8175838594552881 % _timestamp‚1.7845112280199034e+09“çbé¸^Å‚(:Ó „öõÒ‘‰ªË disk.nvme1n1.in‚0) disk./.usagePercent‚76.00436549954647 disk.md0.in‚0 disk.md0.out‚ 139.49609375$ memory_percent‚95.48482894897461& proc.memory.availableMB‚ 739.765625! proc.memory.rssMB‚ 3017.984375 proc.cpu.threads‚17 network.recv‚ 1.166595e+06& disk.nvme0c0n1.out‚26413.271484375( disk.nvme1c1n1.out‚16735.69287109375) proc.memory.percent‚18.42031478881836$ disk./.usageGB‚1166.857021331787 disk.dm-1.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚26413.271484375 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚16735.69287109375 cpu‚25.64792145298173 disk.nvme0n1.in‚ 0.00390625 network.sent‚ 7.677092e+06 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚36177.95458984375! disk.dm-1.out‚6673.3994140625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625àv;"žÆ‚(:“ „öõÒÀÓ™Ì gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.871' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.62366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:A×1ZÇ‚(:Ï “öõÒ±ÒÁË disk.nvme1n1.in‚0 disk.rbd0.out‚0$ memory_percent‚95.48482894897461! proc.memory.rssMB‚ 3017.984375) proc.memory.percent‚18.42031478881836 cpu‚25.647250770580165" disk.dm-1.out‚6673.49658203125' disk.nvme1c1n1.out‚16736.4951171875 disk.md0.out‚ 140.0703125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625" disk.nvme0n1.out‚ 26414.6796875 proc.cpu.threads‚17 disk.md0.in‚0# disk.dm-0.out‚36178.85302734375$ disk.nvme0c0n1.out‚ 26414.6796875 network.recv‚ 1.171742e+06) disk./.usagePercent‚76.00437220834479 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚16736.4951171875 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625& proc.memory.availableMB‚ 739.765625 network.sent‚ 7.740475e+06% disk./.usageGB‚1166.8571243286133TžOžÈ‚(:“ “öõÒ‡çšÌ gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚285.733' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.24433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ›e7\É‚(:Ñ ¢öõÒîàÅË' proc.memory.availableMB‚ 493.2265625) disk./.usagePercent‚76.00437941409113$ disk./.usageGB‚1166.857234954834 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1c1n1.out‚16737.21484375 disk.nvme1n1.in‚0# disk.nvme1n1.out‚16737.21484375 disk.md0.in‚0( disk.nvme0c0n1.out‚26417.15869140625" disk.dm-0.out‚36181.4169921875" disk.dm-1.out‚6673.55908203125$ memory_percent‚96.98958396911621! proc.memory.rssMB‚ 3017.984375) proc.memory.percent‚18.42031478881836 cpu‚25.646698616658043 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.md0.out‚ 140.328125 proc.cpu.threads‚17 network.sent‚ 7.745279e+06 network.recv‚ 1.175066e+06 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚26417.15869140625 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0_ {DœÊ‚(:‘ ¢öõÒœäÌ gpu.0.gpu‚77.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚268.1' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.36666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÊÏ$2Ë- _wandbruntime‚4443  _runtime‚4443á½ßÇÌÁ & lr-AdamW/pg1‚0.0005363721348603061 & lr-AdamW/pg2‚0.0005363721348603061 trainer/global_step‚4949 % _timestamp‚1.7845112704167006e+09 _runtime‚4443.993026607 _step‚404çÔ¥³ÌÍÆ % _timestamp‚1.7845112704167006e+09 _runtime‚4443.993026607 _step‚404 & lr-AdamW/pg1‚0.0005363721348603061 & lr-AdamW/pg2‚0.0005363721348603061 trainer/global_step‚4949”xSdû2Î- _wandbruntime‚4443  _runtime‚4443å*xÝæ'ÏàO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40052949 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚763048740405760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚763048381942272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164641510817280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚752872422180304 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8638305 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5779118538240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚4949 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚757302504206848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40052951 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48955730 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40052949 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5746236198912 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚763048381942272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40052951 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48965873 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚752872439219664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20486997 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5745894774784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚757302487167488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170420629355520 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚752872422180304 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8638305 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5779118538240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚4949 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚757302504206848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40052951 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48955730 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40052949 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5746236198912 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚763048381942272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40052951 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48965873 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚752872439219664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20486997 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5745894774784 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚757302487167488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170420629355520 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚763048740405760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚763048381942272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164641510817280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89036287 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5780342753792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚4949 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48965386 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48975543 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚763197766543360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5738672674229 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚763198125014016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40060744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚757450761669632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89026128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20490703 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5747346305024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164674691087360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89026128 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40060742 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40060742 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ^JDeviceStatsMonâÑèeÛitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚757450778708992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8640039 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚758758184960619 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89036287 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40060744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚757450761669632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚753019869887136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845112710390294e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5747004873728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170455052453376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5780229103104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚763198125014016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5747004873728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20491002 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488f[yö&ÓðM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40060742 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40060742 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚757450778708992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8640039 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚758758184960619 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89036287 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40060744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚757450761669632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚753019869887136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845112710390294e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5747004873728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170455052453376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5780229103104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚763198125014016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5747004873728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20491002 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89036287 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5780342753792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚4949 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48965386 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48975543 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚763197766543360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5738672674229 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚763198125014016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40060744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚757450761669632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89026128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20490703 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5747346305024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164674691087360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89026128 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452–L‰Ýx2Ô- _wandbruntime‚4444  _runtime‚4444ƒØ Õ† epoch‚12 3loss/FBt2/assignment_loss‚0.0003571129586663148 1loss/SRqqt1/detection_loss‚0.7482182781212032 trainer/global_step‚4949 % _timestamp‚1.7845112710401015e+09 _runtime‚4444.615364003 _step‚407 0loss/FRt1/assignment_loss‚1.6917900844709948 2loss/SRqqt1/assignment_loss‚0.5137375229481281 /loss/FRt1/detection_loss‚0.5295746412765787 1loss/SRqqt2/detection_loss‚0.7414198253018361 /loss/FBt1/detection_loss‚0.4353066036366995 /loss/FBt2/detection_loss‚0.5532930331347987 0loss/FRt2/assignment_loss‚0.9080381108366925 %loss/total_loss‚8.225288681621407 2loss/SRqqt2/assignment_loss‚0.5155342662167377 3loss/FBt1/assignment_loss‚0.0014382307385743121 /loss/FRt2/detection_loss‚1.5865809719804973Uk—¾Ö‹ 2loss/SRqqt2/assignment_loss‚0.5155342662167377 3loss/FBt1/assignment_loss‚0.0014382307385743121 /loss/FRt2/detection_loss‚1.5865809719804973 epoch‚12 3loss/FBt2/assignment_loss‚0.0003571129586663148 1loss/SRqqt1/detection_loss‚0.7482182781212032 trainer/global_step‚4949 % _timestamp‚1.7845112710401015e+09 _runtime‚4444.615364003 _step‚407 0loss/FRt1/assignment_loss‚1.6917900844709948 2loss/SRqqt1/assignment_loss‚0.5137375229481281 /loss/FRt1/detection_loss‚0.5295746412765787 1loss/SRqqt2/detection_loss‚0.7414198253018361 /loss/FBt1/detection_loss‚0.4353066036366995 /loss/FBt2/detection_loss‚0.5532930331347987 0loss/FRt2/assignment_loss‚0.9080381108366925 %loss/total_loss‚8.225288681621407—¿lãD[ׂ(:Ð ±öõÒ”ßêË disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚16738.0537109375 disk.nvme1n1.in‚0 disk.md0.in‚0 network.recv‚ 1.17872e+06) disk./.usagePercent‚76.00437717782502 disk.rbd0.in‚0 disk.md0.out‚ 140.77734375 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚36181.9404296875$ memory_percent‚96.98975086212158( proc.memory.availableMB‚ 493.19921875 network.sent‚ 7.763066e+06 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0c0n1.out‚26418.076171875! proc.memory.rssMB‚ 3017.984375) proc.memory.percent‚18.42031478881836 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme1n1.out‚16738.0537109375 cpu‚25.64597932953828% disk./.usageGB‚1166.8572006225586 disk.dm-1.out‚ 6673.83203125 disk.rbd0.out‚0$ disk.nvme0n1.out‚26418.076171875‘y⠞؂(:“ ±öõÒ¸Ê¬Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚285.269' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.08966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=34UÙ‚(:Ê ÀöõÒÿ¬£Ë disk.dm-1.out‚ 6673.8671875' proc.memory.availableMB‚ 493.1953125) disk./.usagePercent‚76.00438338967531 disk.nvme1c1n1.in‚0 disk.md0.out‚141 disk.dm-0.in‚ 0.00390625$ memory_percent‚96.98977470397949) proc.memory.percent‚18.42031478881836 cpu‚25.645275051124326 network.sent‚ 7.811282e+06% disk./.usageGB‚1166.8572959899902 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚26420.02099609375& disk.nvme1c1n1.out‚16738.615234375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625! proc.memory.rssMB‚ 3017.984375 network.recv‚ 1.183627e+06 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚16738.615234375 disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚26420.02099609375 proc.cpu.threads‚17 disk.md0.in‚0# disk.dm-0.out‚36183.91357421875ïò¥6žÚ‚(:“ ÀöõÒ¦†üË gpu.0.gpu‚40.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.874' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.29133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û·Šÿ_Û‚(:Ô ÏöõÒ„ĹË$ memory_percent‚96.98410034179688# proc.memory.availableMB‚494.125 network.sent‚ 7.817355e+06% disk./.usageGB‚1166.8572959899902 disk.dm-0.in‚ 0.00390625! disk.dm-1.out‚6674.0263671875 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚26421.12451171875& disk.nvme0n1.out‚26421.12451171875) proc.memory.percent‚18.42031478881836 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 141.61328125 disk.dm-1.in‚0 cpu‚25.644622218379173 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 1.187012e+06) disk./.usagePercent‚76.00438338967531% disk.nvme1n1.out‚16739.5498046875# disk.dm-0.out‚36184.49951171875' disk.nvme1c1n1.out‚16739.5498046875! proc.memory.rssMB‚ 3017.984375 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0«ÜcÜ‚(:’ ÏöõÒŽ»€Ì gpu.0.gpu‚57.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚289.81' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.60333333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð•|û2Ý- _wandbruntime‚4487  _runtime‚4487,²ýŒÇÞÁ & lr-AdamW/pg2‚0.0005347243159248909 trainer/global_step‚4999 % _timestamp‚1.7845113140501008e+09 _runtime‚4487.625689206 _step‚408 & lr-AdamW/pg1‚0.0005347243159248909Ãá'ÌßÆ trainer/global_step‚4999 % _timestamp‚1.7845113140501008e+09 _runtime‚4487.625689206 _step‚408 & lr-AdamW/pg1‚0.0005347243159248909 & lr-AdamW/pg2‚0.0005347243159248909˜´÷2à- _wandbruntime‚4487  _runtime‚4487 ·pç'ááO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚770517676151296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚764716212274688 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49440382 % _timestamp‚1.7845113140511844e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40442601 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚764716229314048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11962918 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49450525 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚764716212274688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172135488512000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚764716229314048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49450525 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚760243955553904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8725008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20688197 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚4999 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚770517676151296 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172135610296320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40442601 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _step‚409 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89893126 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5801463876608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89882981 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5792706937999 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49440382 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5835067362816 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5793047499273 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚770518034614784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚760243972593264 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166300524321280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5801463876608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20687923 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5801805300736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5801805300736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5834964190720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40442599 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚770518034614784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8725005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚166300542933504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5793047499273 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚770518034614784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚760243972593264 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166300524321280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5801463876608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20687923 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5801805300736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5801805300736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5834964190720 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40442599 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚770518034614784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8725005 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚166300542933504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚770517676151296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚764716212274688 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49440382 % _timestamp‚1.7845113140511844e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40442601 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚764716229314048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11962918 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49450525 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚764716212274688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172135488512000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚764716229314048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49450525 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚760243955553904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8725008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20688197 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184™p8Q2ã- _wandbruntime‚4488  _runtime‚4488(¤žíï&äéM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚770667419223040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5802915406848 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172169902533120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89910589 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40450392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚410 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89900430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20691933 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚760391386221376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5802573975552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚764864486776832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40450394 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49450038 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49460195 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4999 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5836179329536 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5802915406848 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40450392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166333723203584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89910589 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5794155887637 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49460195 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚770667419223040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49450038 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8726739 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚760391403260736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784511314672631e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚766185559148373 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚764864486776832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5802573975552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166333704591360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚770667060752384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4999 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5836179329536 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5802915406848 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40450392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166333723203584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89910589 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5794155887637 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49460195 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚770667419223040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49450038 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8726739 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚760391403260736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784511314672631e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚766185559148373 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚764864486776832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5802573975552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166333704591360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚770667060752384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89900430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20691933 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚760391386221376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5802573975552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚764864486776832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40450394 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49450038 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49460195 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0š쀓Ž2æ- _wandbruntime‚4488  _runtime‚4488|ùt ç… /loss/FRt1/detection_loss‚0.5087833916434539 1loss/SRqqt2/detection_loss‚0.6829527828294915 epoch‚12 2loss/SRqqt1/assignment_loss‚0.5383545460907735 2loss/SRqqt2/assignment_loss‚0.5893862242924187 1loss/SRqqt1/detection_loss‚0.6918107341178803 %loss/total_loss‚8.018745317396466 _step‚411 2loss/FBt2/assignment_loss‚0.012499493439296714 /loss/FBt1/detection_loss‚0.4602277494518569 0loss/FBt2/detection_loss‚0.44705768186599015 _runtime‚4488.248943184 0loss/FRt2/assignment_loss‚0.7743113267926172 2loss/FBt1/assignment_loss‚0.012019842553005193 /loss/FRt2/detection_loss‚1.6160781050644366 trainer/global_step‚4999 % _timestamp‚1.7845113146736696e+09 0loss/FRt1/assignment_loss‚1.6852634392552457é`¾¸èŠ 0loss/FRt1/assignment_loss‚1.6852634392552457 /loss/FRt1/detection_loss‚0.5087833916434539 1loss/SRqqt2/detection_loss‚0.6829527828294915 epoch‚12 2loss/SRqqt1/assignment_loss‚0.5383545460907735 2loss/SRqqt2/assignment_loss‚0.5893862242924187 1loss/SRqqt1/detection_loss‚0.6918107341178803 %loss/total_loss‚8.018745317396466 _step‚411 2loss/FBt2/assignment_loss‚0.012499493439296714 /loss/FBt1/detection_loss‚0.4602277494518569 0loss/FBt2/detection_loss‚0.44705768186599015 _runtime‚4488.248943184 0loss/FRt2/assignment_loss‚0.7743113267926172 2loss/FBt1/assignment_loss‚0.012019842553005193 /loss/FRt2/detection_loss‚1.6160781050644366 trainer/global_step‚4999 % _timestamp‚1.7845113146736696e+09›Þ_é‚(:Ô ÞöõÒºÙùË! proc.memory.rssMB‚ 3017.984375 cpu‚25.64396983259507 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚ 142.01171875 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚36185.95654296875 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚26422.8271484375$ memory_percent‚96.98433876037598 network.recv‚ 1.190109e+06) disk./.usagePercent‚76.00436251785833# disk./.usageGB‚1166.85697555542& disk.nvme1n1.out‚16740.26025390625! disk.dm-1.out‚6674.1240234375 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚26422.8271484375) proc.memory.percent‚18.42031478881836 disk.rbd0.out‚0( disk.nvme1c1n1.out‚16740.26025390625' proc.memory.availableMB‚ 494.0859375 network.sent‚ 7.83312e+06 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0Ñ»Èažê‚(:“ ÞöõÒµÀ¾Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚281.879' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.95966666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G _]ë‚(:Ò íöõÒÔ‚ÃË$ memory_percent‚96.98433876037598) proc.memory.percent‚18.42031478881836 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625 cpu‚25.64333280653465 proc.cpu.threads‚17) disk./.usagePercent‚76.00437991103914& disk.nvme1n1.out‚16740.90478515625 disk.md0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 6674.25390625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0! proc.memory.rssMB‚ 3017.984375 network.sent‚ 7.882627e+06% disk./.usageGB‚1166.8572425842285 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625' proc.memory.availableMB‚ 494.0859375 network.recv‚ 1.19563e+06 disk.md0.out‚ 142.41796875# disk.dm-0.out‚36186.80419921875& disk.nvme0c0n1.out‚26424.025390625$ disk.nvme0n1.out‚26424.025390625( disk.nvme1c1n1.out‚16740.90478515625Á¥Þižì‚(:“ íöõÒ“Ì gpu.0.gpu‚91.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.284' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.09466666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G k™ í‚(:†æPEà üöõÒÛÊÂË disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 7.88851e+06$ disk./.usageGB‚1166.857322692871 disk.md0.in‚0$ disk.nvme0n1.out‚26425.962890625! proc.memory.rssMB‚ 3017.984375 cpu‚25.64269493448216 network.recv‚ 1.199076e+06! disk.nvme1n1.out‚ 17276.296875 disk.md0.out‚ 142.7421875! disk.dm-0.out‚36188.650390625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0c0n1.out‚26425.962890625 disk.nvme1c1n1.in‚0) disk./.usagePercent‚76.00438512899339 disk.dm-1.out‚ 7209.0546875 disk.nvme0n1.in‚ 0.00390625# disk.nvme1c1n1.out‚ 17276.296875# memory_percent‚96.9844102859497( proc.memory.availableMB‚ 494.07421875) proc.memory.percent‚18.42031478881836 proc.cpu.threads‚17×g5>Ÿî‚(:” üöõÒ®œ¡Ì gpu.0.gpu‚38.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.754' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.251333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈÀÞ˜2ï- _wandbruntime‚4530  _runtime‚4530žLyÇðÁ & lr-AdamW/pg2‚0.0005330618863995277 trainer/global_step‚5049 % _timestamp‚1.7845113573926837e+09 _runtime‚4530.969033266 _step‚412 & lr-AdamW/pg1‚0.0005330618863995277кéÌñÆ & lr-AdamW/pg2‚0.0005330618863995277 trainer/global_step‚5049 % _timestamp‚1.7845113573926837e+09 _runtime‚4530.969033266 _step‚412 & lr-AdamW/pg1‚0.0005330618863995277œ7è-~2ò- _wandbruntime‚4530  _runtime‚4530ÐRT”ç'óáO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚773463679078911 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49935177 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚777987328823808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173850412702208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 % _timestamp‚1.7845113573933318e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5857032978432 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚773464036679545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40832249 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20890024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8811708 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5857374402560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚777986970360320 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49935177 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49925034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12078316 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚772129954421248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚772129937381888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5857374402560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚777987328823808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8811705 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚767615505966864 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40832251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20889721 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167959556437504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚777986970360320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚767615488927504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5848530712681 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49925034 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5848190151407 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5890742606848 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40832251 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20889721 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167959556437504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚777986970360320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚767615488927504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5848530712681 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49925034 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5848190151407 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5890742606848 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8811708 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5857374402560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚777986970360320 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49935177 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49925034 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12078316 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚772129954421248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚772129937381888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5857374402560 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452"'v2õ- _wandbruntime‚4531  _runtime‚4531M¬1Rñ&öëM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49944847 trainer/global_step‚5049 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40840042 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8813442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12080249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90774732 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚778136713432064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚772278211884032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90784891 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40840042 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚773612575735381 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚772278228923392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40840044 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167992736707584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚778136713432064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5858484508672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49944847 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90774732 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20893986 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173884708781056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5891860519936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚778136354961408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚767762919594976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5858143077376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5858484508672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12080544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _runtime‚4531.591741158 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173884578615296 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚772278211884032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40840044 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90784891 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20893688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5849639101045 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49934690 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167992718095360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚773612218134635åò-â÷ðM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5891972073472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N×Îü%>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90774732 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚778136713432064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚772278211884032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90784891 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40840042 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚773612575735381 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚772278228923392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40840044 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167992736707584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚778136713432064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5858484508672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49944847 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90774732 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20893986 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173884708781056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5891860519936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚778136354961408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚767762919594976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5858143077376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5858484508672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12080544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _runtime‚4531.591741158 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173884578615296 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚772278211884032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40840044 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90784891 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20893688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5849639101045 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49934690 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167992718095360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚773612218134635 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49944847 trainer/global_step‚5049 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40840042 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8813442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12080249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚175565221203456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚785456264569344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50419829 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚774987039340464 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8898405 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5912602080256 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5946545992704 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚5099 % _timestamp‚1.7845114009680338e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8898408 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚779543679528448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41221899 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169618569941504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚780890695665919 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50419829 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41221899 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5946651261952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚780891053266553 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21091115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚175565097321984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5903673364815 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0Žö ´i „æO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚175565221203456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚785456264569344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50419829 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚774987039340464 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8898405 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5912602080256 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5946545992704 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚5099 % _timestamp‚1.7845114009680338e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8898408 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚779543679528448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41221899 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169618569941504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚780890695665919 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50419829 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41221899 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5946651261952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚780891053266553 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21091115 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚175565097321984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5903673364815 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚785456623032832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12192710 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5912602080256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41221901 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12193001 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21091409 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145¡D—a…‚(:Ö ©÷õÒæäôË network.sent‚ 7.972324e+06) disk./.usagePercent‚75.97344812673653( disk.nvme1c1n1.out‚17278.89111328125& disk.nvme1n1.out‚17278.89111328125 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚26429.748046875% disk./.usageGB‚1166.3823623657227 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚7209.3935546875 disk.nvme0c0n1.in‚ 0.00390625! proc.memory.rssMB‚ 3017.984375 cpu‚25.640965031764264 network.recv‚ 1.211215e+06# disk.dm-0.out‚36192.22705078125 disk.nvme0n1.in‚ 0.00390625$ disk.nvme0n1.out‚26429.748046875$ memory_percent‚96.98805809020996' proc.memory.availableMB‚ 493.4765625 disk.nvme1c1n1.in‚0 disk.md0.out‚ 143.87890625) proc.memory.percent‚18.42031478881836 proc.cpu.threads‚17Jwž†‚(:“ ©÷õÒÄëµÌ gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚279.046' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.01533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©aŸÌ2‡- _wandbruntime‚4575  _runtime‚4575j^§ºñ&ˆëM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50429499 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21095114 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50429499 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5947659195392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5913712179200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91649034 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91659193 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚779691936991232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12195270 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚781039234721643 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8900142 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50419342 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚785606007641088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚418 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚785606007641088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚779691954030592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21095412 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚169651750211584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚779691954030592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚175599390794752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50419342 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0H¹÷Üö&‰ðM FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91659193 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚779691936991232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12195270 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚781039234721643 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8900142 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50419342 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚785606007641088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚418 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚785606007641088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚779691954030592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21095412 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚169651750211584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚779691954030592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚175599390794752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50419342 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50429499 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21095114 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50429499 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5947659195392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5913712179200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91649034 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚786957404635648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171277564833280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12307945 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚782358572714064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚788317712252927 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41611551 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5959156578223 YKDeviceStatsMonitor.on_train_batch_start/reserv¬Šb'$ed_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚786957404635648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5968512606208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50894338 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚792925917241856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚782358555674704 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5968512606208 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5968171182080 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚792925558778368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41611549 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41611549 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50904481 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21293050 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177280049164800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚792925558778368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92505887 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41611551 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8985108 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6002465719296 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6002354158592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚5149 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚421 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845114439958594e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚786957387596288 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚786957387596288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50894338 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12308243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1X¡(æì'–æO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚786957404635648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5968512606208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50894338 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚792925917241856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚782358555674704 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5968512606208 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5968171182080 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚792925558778368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41611549 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41611549 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50904481 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21293050 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177280049164800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚792925558778368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92505887 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41611551 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8985108 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6002465719296 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6002354158592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚5149 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚421 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845114439958594e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚786957387596288 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚786957387596288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50894338 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12308243 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50904481 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177279918991872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚786957404635648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171277564833280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12307945 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚782358572714064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚788317712252927 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41611551 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5959156578223¥v™2—- _wandbruntime‚4618  _runtime‚4618(`«Yñ&˜ëM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12310486 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41619342 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5960605527861 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92523336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6003473366016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚788466608909397 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5969281281024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚787105679137792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92533495 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92523336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50903994 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8986839 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4618.193707536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41619342 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _step‚422 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5969622712320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41619344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5960264966475 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21297043 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177314218469376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50914151 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚793075301850112 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚787105662098432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6003576530944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41619344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚787105662098432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21297328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177314340246528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚782505986342176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92533495 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚793075301850112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚793074943379456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚171310745103360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50903994 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8986842 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5969281281024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92533495 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92523336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50903994 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8986839 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4618.193707536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41619342 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _step‚422 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5969622712320 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41619344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5960264966475 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21297043 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177314218469376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50914151 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚793075301850112 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚787105662098432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6003576530944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41619344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚787105662098432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21297328 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177314340246528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚782505986342176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92533495 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚793075301850112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚793074943379456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚171310745103360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50903994 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8986842 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5969281281024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 VIDeviceStatsMonitUŠ›ÇL or.on_train_batch_end/inactive_split.small_pool.allocated‚12310486 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41619342 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5960605527861 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92523336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6003473366016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚788466608909397 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5969281281024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0¦‚Õy2š- _wandbruntime‚4618  _runtime‚4618à±Ñ ›† 2loss/SRqqt2/assignment_loss‚0.5239053330671958 3loss/FBt2/assignment_loss‚5.098063869940253e-05 0loss/FBt1/detection_loss‚0.48938217222504315 %loss/total_loss‚8.327642317297594 _step‚423 0loss/FRt2/assignment_loss‚0.7177546696522312 /loss/FRt1/detection_loss‚0.5119960053834908 1loss/SRqqt1/detection_loss‚0.7343689379926881 1loss/SRqqt2/detection_loss‚0.7512079239618411 .loss/FRt2/detection_loss‚1.663092552478304 /loss/FBt2/detection_loss‚0.6684759410260164 trainer/global_step‚5149 _runtime‚4618.194312327 0loss/FRt1/assignment_loss‚1.6982223872289897 2loss/SRqqt1/assignment_loss‚0.5656659123365191 3loss/FBt1/assignment_loss‚0.0035195013065751936 epoch‚12 % _timestamp‚1.7845114446190112e+09árCÂœ‹ /loss/FRt1/detection_loss‚0.5119960053834908 1loss/SRqqt1/detection_loss‚0.7343689379926881 1loss/SRqqt2/detection_loss‚0.7512079239618411 .loss/FRt2/detection_loss‚1.663092552478304 /loss/FBt2/detection_loss‚0.6684759410260164 trainer/global_step‚5149 _runtime‚4618.194312327 0loss/FRt1/assignment_loss‚1.6982223872289897 2loss/SRqqt1/assignment_loss‚0.5656659123365191 3loss/FBt1/assignment_loss‚0.0035195013065751936 epoch‚12 % _timestamp‚1.7845114446190112e+09 2loss/SRqqt2/assignment_loss‚0.5239053330671958 3loss/FBt2/assignment_loss‚5.098063869940253e-05 0loss/FBt1/detection_loss‚0.48938217222504315 %loss/total_loss‚8.327642317297594 _step‚423 0loss/FRt2/assignment_loss‚0.7177546696522312§„Ò?¦_‚(:Ô Ö÷õÒÚéæË& disk.nvme1n1.out‚17283.28076171875 disk.dm-0.in‚ 0.00390625$ memory_percent‚96.98920249938965! proc.memory.rssMB‚ 3017.984375 cpu‚25.639274524464188( disk.nvme1c1n1.out‚17283.28076171875 network.recv‚ 1.224832e+06) disk./.usagePercent‚75.97351347540163 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 disk.md0.out‚ 145.09375) proc.memory.percent‚18.42031478881836 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚26448.5693359375 disk.nvme1n1.in‚0 disk.md0.in‚0# disk.dm-0.out‚36212.61474609375' proc.memory.availableMB‚ 493.2890625% disk./.usageGB‚1166.3833656311035 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚26448.5693359375 network.sent‚ 8.081972e+06 disk.dm-1.out‚7209.615234375ÌÆážž‚(:“ Ö÷õÒ¦ï«Ì gpu.0.gpu‚92.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚278.234' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.74466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½üÔeŸ‚(:Ú å÷õÒ„˪Ë disk.nvme1n1.in‚0) proc.memory.percent‚18.42031478881836 proc.cpu.threads‚17 network.sent‚ 8.132347e+06 network.recv‚ 1.230276e+06 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0$ disk./.usageGB‚1166.383659362793 disk.rbd0.out‚0& disk.nvme0n1.out‚26449.73193359375$ memory_percent‚96.99130058288574' proc.memory.availableMB‚ 492.9453125! proc.memory.rssMB‚ 3017.984375 disk.md0.out‚ 145.44921875 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚26449.73193359375& disk.nvme1n1.out‚17283.98095703125 cpu‚25.638669571004396) disk./.usagePercent‚75.97353260790054 disk.md0.in‚0# disk.dm-0.out‚36213.54443359375" disk.dm-1.out‚7209.77783203125( disk.nvme1c1n1.out‚17283.98095703125ìEêdž ‚(:“ å÷õÒÜ«ëË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚269.385' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.79499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“nñc¡‚(:Ø ô÷õÒ¡‚ÃË disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚7209.87158203125 disk.rbd0.out‚0 cpu‚25.638168024927488% disk./.usageGB‚1166.9235000610352( disk.nvme1c1n1.out‚17378.16064453125 disk.nvme1n1.in‚0 disk.md0.out‚ 145.9453125 disk.dm-1.in‚0$ memory_percent‚96.98987007141113 network.sent‚ 8.137199e+06( disk./.usagePercent‚76.0086956561495 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme1n1.out‚17378.16064453125 disk.rbd0.in‚0' proc.memory.availableMB‚ 493.1796875) proc.memory.percent‚18.42031478881836( disk.nvme0c0n1.out‚26561.14892578125 disk.md0.in‚0# disk.dm-0.out‚36418.00048828125! proc.memory.rssMB‚ 3017.984375 proc.cpu.threads‚17 network.recv‚ 1.23342e+06 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚26561.14892578125 disk.nvme1c1n1.in‚0Ü®¨Æž¢‚(:“ ô÷õÒ¸˜‚Ì gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚275.375' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.79166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëì}2£- _wandbruntime‚4660  _runtime‚4660øýAƤÀ & lr-AdamW/pg1‚0.0005279880066820167 & lr-AdamW/pg2‚0.0005279880066820167 trainer/global_step‚5199 $ _timestamp‚1.784511487350754e+09 _runtime‚4660.927045543 _step‚424ÝxšÜ˥Š& lr-AdamW/pg1‚0.0005279880066820167 & lr-AdamW/pg2‚0.0005279880066820167 trainer/global_step‚5199 $ _timestamp‚1.784511487350754e+09 _runtime‚4660.927045543 _step‚424¨WuÖO2¦- _wandbruntime‚4660  _runtime‚4660ئŸç'§áO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚794371112703488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172936596949504 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42001199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚794371112703488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9071805 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚789730089048304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6024081708032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21494734 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚178994752101376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6058283227648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51389133 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚800395211450880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12423225 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6023740283904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42001199 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9071808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172936578337280 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42001201 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845114873514752e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _step‚425 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12422929 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42001201 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51378990 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚178994880177152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51389133 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6058173764096 _runtime‚4660.927122533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚794371129742848 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚794371129742848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6023740283904 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93380189 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21495033 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6014980352905 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚789730106087664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚5199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6024081708032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93390334 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚800394852987392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚795745086440569 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51378990 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚800395211450880 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528¸ÇŠ!ì'¨æO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9071805 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚789730089048304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6024081708032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21494734 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚178994752101376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6058283227648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51389133 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚800395211450880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12423225 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6023740283904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42001199 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9071808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172936578337280 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42001201 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845114873514752e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _step‚425 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12422929 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42001201 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51378990 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚178994880177152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51389133 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6058173764096 _runtime‚4660.927122533 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚794371129742848 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚794371129742848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6023740283904 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93380189 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21495033 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6014980352905 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚789730106087664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚5199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6024081708032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93390334 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚800394852987392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚795745086440569 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51378990 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚800395211450880 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚794371112703488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172936596949504 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42001199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚794371112703488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0©bÒ;æ2©- _wandbruntime‚4661  _runtime‚4661IÿÓì ªêM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6024850382848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51398803 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93407797 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42008992 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51398803 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42008994 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42008992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93397638 trainer/global_step‚5199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6025191814144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12425519 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚795893625496405 $ _timestamp‚1.784511487970461e+09 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21498756 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚789877519715776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6025191814144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚794519387205632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚302 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6059397938688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172969758607360 _step‚426 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6024850382848 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚800544596059136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9073542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42008994 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚800544596059136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179029051283968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚4661.546215321 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51388646 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6015748179883 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0rj®cõ&«ïM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚789877536755136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21499061 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚794519404244992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚794519404244992 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9073539 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12425217 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6016088741269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21498756 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚789877519715776 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6025191814144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚794519387205632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚302 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6059397938688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172969758607360 _step‚426 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6024850382848 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚800544596059136 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9073542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42008994 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚800544596059136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179029051283968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚4661.546215321 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51388646 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6015748179883 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6024850382848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51398803 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93407797 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42008992 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51398803 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42008994 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42008992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93397638 trainer/global_step‚5199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6025191814144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12425519 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚795893625496405 $ _timestamp‚1.784511487970461e+09 P»lK ®† 2loss/SRqqt1/assignment_loss‚0.5145851086487342 2loss/FBt1/assignment_loss‚0.006879355901761256 /loss/FRt1/detection_loss‚0.4740995638934535 /loss/FRt2/detection_loss‚1.7157240413057333 %loss/total_loss‚8.557048208282474 0loss/FRt1/assignment_loss‚1.7599333620876576 2loss/SRqqt2/assignment_loss‚0.5542954200662713 ,loss/FBt2/detection_loss‚0.7062247964651 1loss/SRqqt1/detection_loss‚0.6648848336964023 1loss/SRqqt2/detection_loss‚0.7206373197058931 /loss/FBt1/detection_loss‚0.4861998338265423 epoch‚12 trainer/global_step‚5199 % _timestamp‚1.7845114879714892e+09 _runtime‚4661.546747372 _step‚427 0loss/FRt2/assignment_loss‚0.9493171907125268 2loss/FBt2/assignment_loss‚0.004267381972398445«/ê“_¯‚(:Ô ƒøõÒ˜ä»Ë cpu‚25.6391678614721 network.sent‚ 8.140222e+06' disk.nvme0c0n1.out‚26764.3427734375 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚26764.3427734375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0" proc.memory.rssMB‚ 3018.9609375 network.recv‚ 1.236965e+06 disk.nvme0c0n1.in‚ 0.00390625 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚96.99664115905762' proc.memory.availableMB‚ 492.0703125& disk.nvme1n1.out‚17462.06396484375 disk.md0.out‚ 146.453125 disk.dm-0.in‚ 0.00390625( proc.memory.percent‚18.4262752532959 proc.cpu.threads‚17) disk./.usagePercent‚75.98902844117026% disk./.usageGB‚1166.6215591430664( disk.nvme1c1n1.out‚17462.06396484375 disk.md0.in‚0# disk.dm-0.out‚36703.89111328125! disk.dm-1.out‚7209.9951171875 disk.rbd0.in‚0Þª*ª°‚(:’ ƒøõÒƒì¤Ì gpu.0.gpu‚21.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚284.896' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.96533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>éâ42±- _wandbruntime‚4677  _runtime‚4677þ§;cU3²¸z 7validation_average_jet_accuracy‚0.36740052700042725 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6822916865348816 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5838572382926941 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6620140075683594 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14437241852283478 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7208160161972046 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6526085138320923 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12107968330383301 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_2‚0.1523372083902359 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5027906894683838 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4953703284263611 particle/accuracy_2_of_2‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03808150812983513 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8717129826545715 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14583712816238403 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02034926414489746 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_3_of_3‚0.07617846131324768 particle/accuracy_1_of_2‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8925207853317261 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20632870495319366 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19236266613006592 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29204821586608887 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20176832377910614 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30965879559516907 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6866755485534668 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6637793779373169 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996833324432373 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972710609436035 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20877817273139954 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5854225158691406 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7432247996330261 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4572896361351013 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7349866628646851 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7306514978408813 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5800000429153442 +particle/sensitivity‚0.6202651858329773 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 epoch‚12 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7373393774032593 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_3_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_2_of_5‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7317439913749695 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20496723055839539 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6637793779373169 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7987939119338989 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6947241425514221 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6510180234909058 particle/accuracy_1_of_4‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7210344076156616 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3958333432674408 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9229111671447754 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +jet/accuracy_3_of_4‚0.09615276753902435 trainer/global_step‚5199 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39132389426231384 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5906752347946167 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6977386474609375 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45846763253211975 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9969269633293152 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028899459168314934 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7"Purity/*FRt1SRqqt0FBt/event_purity‚0.38863605260849 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9946363568305969 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.955255389213562 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884463548660278 _runtime‚4677.424872544 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9963346123695374 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6581096053123474 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19236266613006592 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9051020741462708 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13779987394809723 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22463108599185944 *jet/accuracy_1_of_2‚0.7920100092887878 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12662407755851746 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7478941679000854 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1849139779806137 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.164166659116745 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03808150812983513 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7332525253295898 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6967616677284241 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6609535813331604 )loss/val_total_loss‚8.050582688691351 +jet/accuracy_1_of_1‚0.32266783714294434 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6609237194061279 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9885153770446777 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912253022193909 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6161651015281677 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7739566564559937 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18337243795394897 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20805886387825012 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4940078854560852 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019010309129953384 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7605443596839905 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03827062249183655 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1837652325630188 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03932743892073631 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25935089588165283 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6434746980667114 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_1‚1 % _timestamp‚1.7845115038488064e+09 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9076626896858215 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7148746848106384 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_4_of_5‚0 particle/accuracy_2_of_5‚1 _step‚428 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04017137736082077 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20612534880638123 particle/accuracy_3_of_4‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952335357666016 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6674599647521973 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15806853771209717 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18340414762496948 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.9051020741462708 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6265920996665955 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8252268433570862 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 'particle/f_score‚0.6429459452629089 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6656224131584167 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18288716673851013 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7"Purity/0FRt0SRqqt2FBt/event_purity‚0.98065584897995 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1694483906030655 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 /particle/accuracy_4_of_4‚0.9405600428581238 (particle/accuracy‚0.8520585894584656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963668584823608 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16279785335063934 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 7"Purity/*FRt*SRqqt0FBt/event_purity‚0.27108234167099 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010317697189748287 *jet/accuracy_1_of_3‚0.9249529838562012 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6916424632072449 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5847803950309753 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/Œ&9•é 2FRt2SRqqt1FBt/event_proportion‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4705873131752014 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8907595872879028 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9687905311584473 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4871879816055298 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03872974216938019 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1687212735414505 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7555123567581177 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19330063462257385 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 (jet/accuracy_2_of_3‚0.54914790391922 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7006483674049377 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03959140554070473 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7769425511360168 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15982647240161896 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.659927248954773 +particle/specificity‚0.9154555201530457 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9903279542922974 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03828698396682739 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7543625831604004 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7987939119338989 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8873392939567566 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9063868522644043 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6750487685203552K´Ÿ1C=³½z ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.164166659116745 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03808150812983513 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7332525253295898 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6967616677284241 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6609535813331604 )loss/val_total_loss‚8.050582688691351 +jet/accuracy_1_of_1‚0.32266783714294434 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6609237194061279 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9885153770446777 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912253022193909 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6161651015281677 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7739566564559937 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18337243795394897 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20805886387825012 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4940078854560852 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019010309129953384 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7605443596839905 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03827062249183655 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1837652325630188 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03932743892073631 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25935089588165283 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6434746980667114 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_1‚1 % _timestamp‚1.7845115038488064e+09 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9076626896858215 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7148746848106384 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_4_of_5‚0 particle/accuracy_2_of_5‚1 _step‚428 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04017137736082077 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20612534880638123 particle/accuracy_3_of_4‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952335357666016 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6674599647521973 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15806853771209717 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18340414762496948 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.9051020741462708 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6265920996665955 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8252268433570862 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 'particle/f_score‚0.6429459452629089 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6656224131584167 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18288716673851013 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7"Purity/0FRt0SRqqt2FBt/event_purity‚0.98065584897995 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1694483906030655 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 /particle/accuracy_4_of_4‚0.9405600428581238 (particle/accuracy‚0.8520585894584656 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963668584823608 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16279785335063934 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 7"Purity/*FRt*SRqqt0FBt/event_purity‚0.27108234167099 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010317697189748287 *jet/accuracy_1_of_3‚0.9249529838562012 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6916424632072449 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5847803950309753 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4705873131752014 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8907595872879028 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9687905311584473 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4871879816055298 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03872974216938019 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1687212735414505 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7555123567581177 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19330063462257385 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 (jet/accuracy_2_of_3‚0.54914790391922 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7006483674049377 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03959140554070473 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7769425511360168 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15982647240161896 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.659927248954773 +particle/specificity‚0.9154555201530457 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9903279542922974 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03828698396682739 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7543625831604004 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7987939119338989 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8873392939567566 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9063868522644043 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6750487685203552 7validation_average_jet_accuracy‚0.36740052700042725 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6822916865348816 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5838572382926941 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6620140075683594 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14437241852283478 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7208160161972046 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6526085138320923 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12107968330383301 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_2‚0.1523372083902359 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5027906894683838 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4953703284263611 particle/accuracy_2_of_2‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03808150812983513 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8717129826545715 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14583712816238403 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02034926414489746 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_3_of_3‚0.07617846131324768 particle/accuracy_1_of_2‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8925207853317261 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20632870495319366 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19236266613006592 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29204821586608887 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20176832377910614 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30965879559516907 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6866755485534668 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6637793779373169 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.996833324432373 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972710609436035 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20877817273139954 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5854225158691406 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7432247996330261 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4572896361351013 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7349866628646851 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7306514978408813 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5800000429153442 +particle/sensitivity‚0.6202651858329773 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 epoch‚12 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7373393774032593 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_3_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_2_of_5‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7317439913749695 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20496723055839539 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6637793779373169 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7987939119338989 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6947241425514221 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6510180234909058 particle/accuracy_1_of_4‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7210344076156616 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3958333432674408 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9229111671447754 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +jet/accuracy_3_of_4‚0.09615276753902435 trainer/global_step‚5199 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39132389426231384 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5906752347946167 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6977386474609375 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45846763253211975 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9969269633293152 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028899459168314934 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7"Purity/*FRt1SRqqt0FBt/event_purity‚0.38863605260849 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9946363568305969 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.955255389213562 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9884463548660278 _runtime‚4677.424872544 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9963346123695374 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6581096053123474 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19236266613006592 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9051020741462708 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13779987394809723 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22463108599185944 *jet/accuracy_1_of_2‚0.7920100092887878 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12662407755851746 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7478941679000854 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_3_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1849139779806137¬¬ò`´‚(:Õ ’øõÒŒãÈË network.sent‚ 8.203607e+06) disk./.usagePercent‚75.98901949610584 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0( disk.nvme0c0n1.out‚26775.13330078125 cpu‚25.64511031539238 network.recv‚ 1.242022e+06% disk./.usageGB‚1166.6214218139648 disk.md0.out‚ 147.0078125 disk.dm-1.in‚0! disk.dm-1.out‚7455.4560546875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625$ memory_percent‚95.49238681793213& disk.nvme1c1n1.out‚17708.974609375 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚26775.13330078125( proc.memory.availableMB‚ 738.52734375" proc.memory.rssMB‚ 3018.9609375( proc.memory.percent‚18.4262752532959 proc.cpu.threads‚17$ disk.nvme1n1.out‚17708.974609375# disk.dm-0.out‚36714.95068359375 disk.rbd0.out‚0`ï蜵‚(:‘ ’øõÒ·ý´Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.876' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.62533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊyyL^¶‚(:Ó ¡øõÒ¦§œË% disk.nvme1n1.out‚17709.4560546875 disk.md0.in‚0( proc.memory.availableMB‚ 738.30859375 cpu‚25.64454613563616 disk.dm-1.in‚0# memory_percent‚95.4937219619751 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 147.2890625" disk.dm-1.out‚7455.56494140625( proc.memory.percent‚18.4262752532959 proc.cpu.threads‚17 network.recv‚ 1.247402e+06) disk./.usagePercent‚75.98902322321601 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625$ disk.nvme0n1.out‚26775.908203125 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚36715.47802734375" proc.memory.rssMB‚ 3018.9609375 network.sent‚ 8.260378e+06% disk./.usageGB‚1166.6214790344238& disk.nvme0c0n1.out‚26775.908203125' disk.nvme1c1n1.out‚17709.4560546875gAÀž·‚(:“ ¡øõÒñ¿×Ë gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚276.608' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.20266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷skb¸‚(:× °øõÒ„ ©Ë% disk.nvme1n1.out‚17958.9814453125" proc.memory.rssMB‚ 3018.9609375 cpu‚25.644033450698014 network.sent‚ 8.264104e+06) disk./.usagePercent‚75.97354254686101 disk.md0.in‚0# disk.dm-0.out‚36737.71337890625 disk.nvme0n1.in‚ 0.00390625$ memory_percent‚95.49221992492676 network.recv‚ 1.250655e+06 disk.md0.out‚ 147.63671875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚26794.7900390625' proc.memory.availableMB‚ 738.5546875 proc.cpu.threads‚17% disk./.usageGB‚1166.3838119506836" disk.dm-1.out‚7700.99853515625 disk.rbd0.out‚0' disk.nvme1c1n1.out‚17958.9814453125 disk.nvme1n1.in‚0( proc.memory.percent‚18.4262752532959 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0% disk.nvme0n1.out‚26794.7900390625 disk.nvme1c1n1.in‚0VˆÏ&œ¹‚(:‘ °øõÒ®ñüË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚285.706' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.23533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó½F2º- _wandbruntime‚4720  _runtime‚4720pnÜÅ»¿ _runtime‚4720.433942515 _step‚429 % lr-AdamW/pg1‚0.000526268211825162 % lr-AdamW/pg2‚0.000526268211825162 trainer/global_step‚5249 % _timestamp‚1.7845115468583608e+097'ŒØÊ¼Ä % lr-AdamW/pg1‚0.000526268211825162 % lr-AdamW/pg2‚0.000526268211825162 trainer/global_step‚5249 % _timestamp‚1.7845115468583608e+09 _runtime‚4720.433942515 _step‚429­|h\2½- _wandbruntime‚4720  _runtime‚4720(D6E'¾àO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94458798 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12573804 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚799022878430128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6090767415215 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚803727198725632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12573530 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚809826808910336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6099627224064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6099627224064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51955284 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9162621 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚805113305284069 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6090426853941 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94468943 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51945141 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚803727181686272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚4720.434753747 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6134881421312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚809826808910336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚799022895469488 % _timestamp‚1.7845115468593175e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42513659 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6099968648192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚809827167373824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174618189115904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚809827167373824 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21736428 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21736151 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚803727181686272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚5249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚803727198725632 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180753070537216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51955284 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6134784540672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42513659 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9162624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180752955044352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51945141 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174618170503680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 2456íóÑlV6038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180ù»)Àë'¿åO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚803727181686272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚4720.434753747 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6134881421312 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚809826808910336 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚799022895469488 % _timestamp‚1.7845115468593175e+09 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42513659 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6099968648192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚809827167373824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174618189115904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚809827167373824 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21736428 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21736151 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚803727181686272 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚5249 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚803727198725632 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180753070537216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51955284 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6134784540672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42513659 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9162624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180752955044352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51945141 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174618170503680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94458798 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12573804 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚799022878430128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6090767415215 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚803727198725632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12573530 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚809826808910336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6099627224064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6099627224064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51955284 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9162621 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚805113305284069 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6090426853941 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94468943 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51945141 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303®«š™M2À- _wandbruntime‚4721  _runtime‚4721äêésð&ÁêM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21740238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6135897653248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚805262201940539 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42521452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6091875803579 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51954797 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94476247 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 4721.04867955 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12576165 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42521450 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180787370204160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94486406 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚809976551982080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚799170326136960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21740523 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 4721.04867955 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12576165 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42521450 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180787370204160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94486406 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚809976551982080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚799170326136960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21740523 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21740238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6135897653248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚805262201940539 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42521452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6091875803579 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51954797 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94476247 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚817296103119360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95343245 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176277184007680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚817296103119360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811140906793472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚4763.039630474 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚817296461582848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6155196325888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6155196325888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6155537750016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52429793 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚806394411803728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811140923832832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚806394428843088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182467905490944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚817296103119360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95343245 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176277184007680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚817296103119360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811140906793472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚4763.039630474 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚817296461582848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6155196325888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6155196325888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6155537750016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52429793 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚806394411803728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811140923832832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚806394428843088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182467905490944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚435 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚811289181295616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52449606 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9251055 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52439449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12691689 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚811289198334976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6156647856128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52439449 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176310382889984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6191941033472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42911100 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95350549 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12691971 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚811289181295616 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95360708 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95360708 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21942744 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚812688860926801 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42911102 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚817445846191104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176310364277760 trainer/global_step‚5299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6191833674240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚806541859510560 _runtime‚4763.657677056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21943029 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95350549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6156306424832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚811289198334976 UEDeviceStatsMonitor.on_train_batch_end/allocated_%x%bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6156306424832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚817445846191104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6147018455601 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0°»Üßö&ÔðM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176310364277760 trainer/global_step‚5299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6191833674240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚806541859510560 _runtime‚4763.657677056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21943029 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95350549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6156306424832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚811289198334976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6156306424832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚817445846191104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6147018455601 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚817445487720448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42911102 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6147359016987 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚435 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚811289181295616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52449606 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9251055 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52439449 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12691689 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚811289198334976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6156647856128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52439449 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176310382889984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6191941033472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42911100 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95350549 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12691971 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚811289181295616 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95360708 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95360708 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21942744 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚812688860926801 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42911102 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚817445846191104³U ØX2Õ- _wandbruntime‚4763  _runtime‚4763ìx6\ Ö… 0loss/FRt2/assignment_loss‚0.7920124958916969 2loss/SRqqt1/assignment_loss‚0.5006501828931021 2loss/SRqqt2/assignment_loss‚0.5579325430409617 4loss/FBt2/assignment_loss‚0.00047051387352465516 /loss/FRt1/detection_loss‚0.4883812202134484 2loss/FBt1/assignment_loss‚0.023263817683833662 1loss/SRqqt1/detection_loss‚0.7241492208430074 /loss/FBt1/detection_loss‚0.4094462911630722 1loss/SRqqt2/detection_loss‚0.7059660394022284 %loss/total_loss‚8.376380559527519 trainer/global_step‚5299 $ _timestamp‚1.784511590082978e+09 _runtime‚4763.658278307 _step‚436 0loss/FRt1/assignment_loss‚1.8267892327112116 /loss/FRt2/detection_loss‚1.6860076166561753 /loss/FBt2/detection_loss‚0.6613113851552563 epoch‚13Áی׊ 1loss/SRqqt1/detection_loss‚0.7241492208430074 /loss/FBt1/detection_loss‚0.4094462911630722 1loss/SRqqt2/detection_loss‚0.7059660394022284 %loss/total_loss‚8.376380559527519 trainer/global_step‚5299 $ _timestamp‚1.784511590082978e+09 _runtime‚4763.658278307 _step‚436 0loss/FRt1/assignment_loss‚1.8267892327112116 /loss/FRt2/detection_loss‚1.6860076166561753 /loss/FBt2/detection_loss‚0.6613113851552563 epoch‚13 0loss/FRt2/assignment_loss‚0.7920124958916969 2loss/SRqqt1/assignment_loss‚0.5006501828931021 2loss/SRqqt2/assignment_loss‚0.5579325430409617 4loss/FBt2/assignment_loss‚0.00047051387352465516 /loss/FRt1/detection_loss‚0.4883812202134484 2loss/FBt1/assignment_loss‚0.023263817683833662´›\÷AaØ‚(:Ö ìøõÒöÀýË! disk.dm-1.out‚7701.4990234375 disk.nvme1n1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1166.6854667663574 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚27007.93212890625 disk.md0.out‚ 149.07421875& disk.nvme0n1.out‚27007.93212890625 network.recv‚ 1.265236e+06) disk./.usagePercent‚75.99319112628936 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0% proc.memory.availableMB‚ 492.15625 disk.rbd0.out‚0( disk.nvme1c1n1.out‚18046.56298828125& disk.nvme1n1.out‚18046.56298828125 disk.md0.in‚0# memory_percent‚96.9961166381836" proc.memory.rssMB‚ 3018.9609375( proc.memory.percent‚18.4262752532959 cpu‚25.641802468241373 network.sent‚ 8.352913e+06" disk.dm-0.out‚37034.8056640625²ÌÁÙ‚(:’ ìøõÒÐȽÌ gpu.0.gpu‚45.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚289.51' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.50333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ª©]Ú‚(:Ò ûøõÒÌžË network.recv‚ 1.270849e+06) disk./.usagePercent‚75.99324877226009 disk.md0.in‚0 disk.md0.out‚ 149.6484375( disk.nvme0c0n1.out‚27036.46533203125( disk.nvme1c1n1.out‚18048.01513671875# memory_percent‚96.9961166381836 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 cpu‚25.641211642712666$ disk./.usageGB‚1166.686351776123" disk.dm-0.out‚37063.3603515625 disk.dm-1.out‚ 7701.69921875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625% proc.memory.availableMB‚ 492.15625 proc.cpu.threads‚17 network.sent‚ 8.402554e+06 disk.nvme1n1.in‚0& disk.nvme1n1.out‚18048.01513671875 disk.dm-1.in‚0& disk.nvme0n1.out‚27036.46533203125" proc.memory.rssMB‚ 3018.9609375( proc.memory.percent‚18.4262752532959ìn뚜ۂ(:‘ ûøõÒ¾ÎñË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.386' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.46200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05A`\Ü‚(:Ñ ŠùõÒµÆË disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚27163.08056640625 disk.md0.in‚0# disk.dm-0.out‚37254.02587890625" proc.memory.rssMB‚ 3018.9609375 proc.cpu.threads‚17( disk./.usagePercent‚75.9890736634404% disk./.usageGB‚1166.6222534179688 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0 network.sent‚ 8.409387e+06 disk.nvme0n1.in‚ 0.00390625% disk.nvme1c1n1.out‚18113.51171875 disk.dm-0.in‚ 0.00390625 disk.dm-1.out‚ 7701.73828125( proc.memory.availableMB‚ 492.15234375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚27163.08056640625 disk.nvme1n1.in‚0# disk.nvme1n1.out‚18113.51171875 disk.md0.out‚ 150.31640625# memory_percent‚96.9961404800415( proc.memory.percent‚18.4262752532959 cpu‚25.640612651877674 network.recv‚ 1.274108e+06&n¨ žÝ‚(:“ ŠùõÒ°á‰Ì gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.544' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.51466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ÊçŸ2Þ- _wandbruntime‚4806  _runtime‚4806p+`XÇßÁ & lr-AdamW/pg1‚0.0005227865310636405 & lr-AdamW/pg2‚0.0005227865310636405 trainer/global_step‚5349 % _timestamp‚1.7845116325868251e+09 _runtime‚4806.162425029 _step‚437öÛßÌàÆ & lr-AdamW/pg1‚0.0005227865310636405 & lr-AdamW/pg2‚0.0005227865310636405 trainer/global_step‚5349 % _timestamp‚1.7845116325868251e+09 _runtime‚4806.162425029 _step‚437µ±ÛV2á- _wandbruntime‚4806  _runtime‚4806}õ²eæ'âàO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚813765962216688 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96207402 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚818554648940032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96217547 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6210765427712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845116325878594e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚818554631900672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9336021 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52924588 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6246544299520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚824765397328384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚5349 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177936197511680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52914445 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43292959 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43292959 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12804271 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚824765397328384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177936216123904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43292957 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚824765755791872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184182865692672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚824765755791872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52914445 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚818554631900672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6210765427712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12804550 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚819967696058719 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6246649568768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Óny‰SãåO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177936197511680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52914445 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43292959 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43292959 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12804271 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚824765397328384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177936216123904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43292957 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚824765755791872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inac0ÍfA˜tive_split_bytes.all.allocated‚184182865692672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚824765755791872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52914445 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚818554631900672 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6210765427712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12804550 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚819967696058719 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6246649568768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚813765962216688 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96207402 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚818554648940032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96217547 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6210765427712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845116325878594e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚818554631900672 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9336021 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52924588 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6246544299520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚824765397328384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚5349¶ý–›2ä- _wandbruntime‚4807  _runtime‚4807Õ%P…ñ&åëM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚818702923442176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43300750 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _step‚439 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43300750 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96235010 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6211875526656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177969396393984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚820115877513809 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184217037028864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52934258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 % _timestamp‚1.7845116335173695e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚820116235114555 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96224851 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52934258 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚824915140400128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96235010 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12806753 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚4807.093131477 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52924101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22144226 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9337755 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚818702906402816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6211875526656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184217163000320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6247766606336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12806471 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96235010 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6211875526656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177969396393984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚820115877513809 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184217037028864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52934258 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 % _timestamp‚1.7845116335173695e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚820116235114555 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96224851 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52934258 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚824915140400128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96235010 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12806753 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚4807.093131477 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52924101 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22144226 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9337755 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚818702906402816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6211875526656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184217163000320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6247766606336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12806471 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179595229627904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 _step‚442 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53399097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9422721 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚185897728347648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMoni`?!tor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚179595211015680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43682607 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚825968374047232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53409240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12919779 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845116765968587e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43682609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12920077 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6257217055439 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6266675953664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚185897604466176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6302498719744 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚832234691537408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚821137495590288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6302393450496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚825968357007872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22342500 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚5399 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6266675953664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97091849 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚832234691537408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53409240 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53399097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚832235050000896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚821137478550928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚825968357007872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚825968374047232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚827394712645727 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97081704 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43682609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12920077 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6257217055439 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6266675953664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚185897604466176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6302498719744 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚832234691537408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚821137495590288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6302393450496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚825968357007872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22342500 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚5399 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6266675953664 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97091849 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚832234691537408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53409240 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53399097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚832235050000896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚821137478550928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚825968357007872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚825968374047232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚827394712645727 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97081704 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179595229627904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 _step‚442 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53399097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9422721 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚185897728347648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚179595211015680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43682607 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚825968374047232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53409240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12919779 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452º·RI2ö- _wandbruntime‚4850  _runtime‚4850Oþ¡¾ñ&÷ëM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6267444628480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6267786059776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚826116648549376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53418910 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚832384434609152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53418910 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179628409897984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6303612775424 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚821284909218400 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97109312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43690402 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22346546 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12922091 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6258325443803 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚821284926257760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚826116631510016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22346839 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚5399 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12922381 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9424458 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9424455 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚832384076138496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6267444628480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6257984882417 % _timestamp‚1.7845116772189367e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53408753 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚4850.794689793 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97109312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43690402 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22346546 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12922091 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6258325443803 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚821284926257760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚826116631510016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22346839 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚5399 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12922381 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9424458 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9424455 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚832384076138496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6267444628480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMoE~ fnitor.on_train_batch_end/inactive_split.all.current‚293 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6257984882417 % _timestamp‚1.7845116772189367e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53408753 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚4850.794689793 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53418910 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚832384434609152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53418910 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179628409897984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6303612775424 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚821284909218400 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 P"aþ‚(:Ö ÕùõÒ˜„ËË disk.rbd0.in‚0( disk.nvme0c0n1.out‚27357.94091796875& disk.nvme0n1.out‚27357.94091796875) disk./.usagePercent‚76.00457769635251 disk.dm-1.out‚ 7702.29296875 disk.rbd0.out‚0 disk.nvme1n1.in‚0" proc.memory.rssMB‚ 3018.9609375( proc.memory.percent‚18.4262752532959 cpu‚25.638404938682122 disk.md0.in‚0 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚18181.38427734375 proc.cpu.threads‚17 network.recv‚ 1.294765e+06! disk.dm-0.out‚37511.685546875 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚18181.38427734375$ memory_percent‚96.99466228485107( proc.memory.availableMB‚ 492.39453125 network.sent‚ 8.546629e+06$ disk./.usageGB‚1166.860279083252 disk.md0.out‚ 152.41015625 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0{lÖžÿ‚(:“ ÕùõÒÇ—¡Ì gpu.0.gpu‚91.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚256.439' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.47966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ED&öY€‚(:Î äùõÒij’Ë disk.nvme1n1.in‚0& disk.nvme1n1.out‚18182.75341796875 cpu‚25.637731861569197 network.recv‚ 1.2981e+06% disk./.usageGB‚1166.8597946166992( disk.nvme1c1n1.out‚18182.75341796875 disk.md0.in‚0 disk.md0.out‚152.75$ memory_percent‚96.99466228485107( proc.memory.availableMB‚ 492.39453125" proc.memory.rssMB‚ 3018.9609375 network.sent‚ 8.552551e+06 disk.rbd0.out‚0& disk.nvme0n1.out‚27362.71630859375 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625! disk.dm-0.out‚37517.064453125 disk.dm-1.in‚0( proc.memory.percent‚18.4262752532959 disk.nvme0c0n1.in‚ 0.00390625 disk.dm-1.out‚ 7702.3359375 proc.cpu.threads‚17) disk./.usagePercent‚76.00454614015302 disk.rbd0.in‚0( disk.nvme0c0n1.out‚27362.71630859375 disk.nvme0n1.in‚ 0.00390625‡ÐÄœ‚(:‘ äùõÒú™æË gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚259.9' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.63333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l%?y2‚- _wandbruntime‚4893  _runtime‚4893ág™áǃÁ & lr-AdamW/pg1‚0.0005192494757979478 & lr-AdamW/pg2‚0.0005192494757979478 trainer/global_step‚5449 % _timestamp‚1.7845117202784994e+09 _runtime‚4893.855016149 _step‚445À8>Ì„Æ trainer/global_step‚5449 % _timestamp‚1.7845117202784994e+09 _runtime‚4893.855016149 _step‚445 & lr-AdamW/pg1‚0.0005192494757979478 & lr-AdamW/pg2‚0.0005192494757979478½—äUc2…- _wandbruntime‚4893  _runtime‚4893Ãðý1ç'†áO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6322245055488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚839703985746432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5449 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53893892 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚834821729232735 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6358332149760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚839704344209920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187612447205888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6321903631360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44072259 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13034509 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97956006 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44072257 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187612575281664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44072259 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6322245055488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181254224519680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53883749 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9509421 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53883749 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚833382099154432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚839704344209920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚833382099154432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 % _timestamp‚1.7845117202793713e+09 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚828509028963888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9509424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13034222 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97966151 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚833382082115072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22543643 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚833382082115072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚834821371632101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6312359707573 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53893892 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6321903631360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6312700268847 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44072257 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚839703985746432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚828509011924528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚839703985746432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚828509011924528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6358332149760 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚839704344209920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187612447205888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6321903631360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44072259 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13034509 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97956006 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44072257 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187612575281664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44072259 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6322245055488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181254224519680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53883749 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9509421 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53883749 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚833382099154432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚839704344209920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚833382099154432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 % _timestamp‚1.7845117202793713e+09 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚828509028963888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9509424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13034222 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97966151 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚833382082115072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22543643 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799¾%#9b2ˆ- _wandbruntime‚4894  _runtime‚4894Ëô2ð‰ëM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large7(ä&_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚839853728818176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13036511 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6359448644608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53893405 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44080052 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6313808657211 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97983614 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44080052 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6313468095825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6323013730304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6323013730304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22547666 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97973455 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚828656459631360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚834970268288571 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97973455 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚839853728818176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6323355161600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187646872046592 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚834969910687825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚833530356617216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53903562 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9511158 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚5449 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44080050 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22547954 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187646739783680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚4894.479356357 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚839853370347520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚833530373656576 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚828656442592000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496pÅ©ö&ŠðM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44080050 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9511155 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚834969910687825 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚833530356617216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53903562 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9511158 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚5449 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44080050 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22547954 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187646739783680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚4894.479356357 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚839853370347520 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚833530373656576 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚828656442592000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚839853728818176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13036511 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6359448644608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53893405 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44080052 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6313808657211 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97983614 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44080052 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6313468095825 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6323013730304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6323013730304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22547666 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97973455 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚828656459631360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚834970268288571 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97973455 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚839853728818176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6323355161600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187646872046592 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚847173279955456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182913256635904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44461909 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54368401 _step‚450 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98840453 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22745730 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4937.133843287 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13149888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13149609 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44461907 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98830308 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚842248745819743 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚847173279955456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6377472733184 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚842248388219109 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6367842920981 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845117635583632e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54378544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚840795807222272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6414136600064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚847173638418944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_star_ÝŠ t/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚5499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚840795807222272 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚835880545298128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6368183482255 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6377472733184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44461907 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9596124 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189327273548800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9596121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44461909 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚840795824261632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6377814157312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54368401 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189327393235968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚847173638418944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647Q–ì'™æO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54368401 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189327393235968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚847173638418944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182913238023680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚847173279955456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182913256635904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44461909 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54368401 _step‚450 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98840453 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22745730 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4937.133843287 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13149888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13149609 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44461907 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98830308 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚842248745819743 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚847173279955456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6377472733184 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚842248388219109 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6367842920981 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845117635583632e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54378544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚840795807222272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6414136600064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚847173638418944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚5499 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚840795807222272 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚835880545298128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6368183482255 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6377472733184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44461907 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9596124 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189327273548800 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9596121 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44461909 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚840795824261632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6377814157312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488Â9UÔ$2š- _wandbruntime‚4937  _runtime‚4937´9¾øñ&›ëM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44469702 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚840944098763776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189361689084928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6369291870619 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98847757 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6378582832128 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚840944081724416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189361569404928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182946418293760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9597855 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6368951309233 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚5499 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6415252178944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6378924263424 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54378057 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚836027993004960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54388214 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚842397284875579 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44469700 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 _step‚451 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22750002 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22749716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98857916 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44469702 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 % _timestamp‚1.7845117641835089e+09 _runtime‚4937.759276706O‡EGD$œðM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚847322664556544 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6378582832128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚840944081724416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189361569404928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182946418293760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9597855 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6368951309233 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚5499 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6415252178944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6378924263424 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54378057 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚836027993004960 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54388214 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚842397284875579 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44469700 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 _step‚451 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22750002 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22749716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98857916 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44469702 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 % _timestamp‚1.7845117641835089e+09 _runtime‚4937.759276706 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44469702 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚840944098763776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189361689084928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6369291870619 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98847757 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6378582832128 F2- _wandbruntime‚4937  _runtime‚49378î % _timestamp‚1.7845117641845982e+09 _step‚452 /loss/FRt2/detection_loss‚1.5631335870124363 1loss/SRqqt2/detection_loss‚0.6876015357584827 /loss/FBt2/detection_loss‚0.7959594373027549 0loss/FRt2/assignment_loss‚0.8159542697063391 2loss/SRqqt1/assignment_loss‚0.5212509104399461 2loss/FBt1/assignment_loss‚0.010313080535123304 .loss/FBt2/assignment_loss‚0.06603695318147 0loss/SRqqt1/detection_loss‚0.741226602303049 _runtime‚4937.759892047 0loss/FRt1/assignment_loss‚1.6777451376366004 2loss/SRqqt2/assignment_loss‚0.5835049523642024 0loss/FBt1/detection_loss‚0.45643310280678223 %loss/total_loss‚8.393127903910706 trainer/global_step‚5499 /loss/FRt1/detection_loss‚0.4739683348635194 epoch‚13ñ7 Ÿ… 1loss/SRqqt2/detection_loss‚0.6876015357584827 /loss/FBt2/detection_loss‚0.7959594373027549 0loss/FRt2/assignment_loss‚0.8159542697063391 2loss/SRqqt1/assignment_loss‚0.5212509104399461 2loss/FBt1/assignment_loss‚0.010313080535123304 .loss/FBt2/assignment_loss‚0.06603695318147 0loss/SRqqt1/detection_loss‚0.741226602303049 _runtime‚4937.759892047 0loss/FRt1/assignment_loss‚1.6777451376366004 2loss/SRqqt2/assignment_loss‚0.5835049523642024 0loss/FBt1/detection_loss‚0.45643310280678223 %loss/total_loss‚8.393127903910706 trainer/global_step‚5499 /loss/FRt1/detection_loss‚0.4739683348635194 epoch‚13 % _timestamp‚1.7845117641845982e+09 _step‚452 /loss/FRt2/detection_loss‚1.5631335870124363Äe½‘Ú] ‚(:Ò  úõÒ¡Ó•Ì disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚27370.17041015625 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚7702.79052734375 disk.rbd0.in‚0$ disk.nvme1n1.out‚18186.013671875 cpu‚25.635553619759076! disk.dm-0.out‚37523.728515625& disk.nvme1c1n1.out‚18186.013671875 disk.md0.in‚0$ memory_percent‚96.99530601501465 proc.cpu.threads‚17 network.sent‚ 8.628229e+06) disk./.usagePercent‚76.00459012005311% disk./.usageGB‚1166.8604698181152 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0' proc.memory.availableMB‚ 492.2890625 network.recv‚ 1.313838e+06( disk.nvme0c0n1.out‚27370.17041015625 disk.nvme1n1.in‚0 disk.md0.out‚ 154.40625" proc.memory.rssMB‚ 3018.9609375( proc.memory.percent‚18.42627525329594d¶‘¡‚(:†  úõÒËÙíÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚287.301' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.767 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lͦ`¢‚(:Õ ¯úõÒêØÛË disk.md0.out‚ 154.8984375( disk.nvme1c1n1.out‚18187.06396484375 cpu‚25.635953034698673 proc.cpu.threads‚17 network.sent‚ 8.692486e+06 network.recv‚ 1.319314e+06$ disk./.usageGB‚1166.860408782959 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚27371.18115234375 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0& disk.nvme1n1.out‚18187.06396484375# memory_percent‚96.9963788986206( proc.memory.availableMB‚ 492.11328125" proc.memory.rssMB‚ 3018.9609375( proc.memory.percent‚18.4262752532959) disk./.usagePercent‚76.00458614446892 disk.dm-0.in‚ 0.00390625! disk.dm-0.out‚37524.525390625 disk.dm-1.out‚7703.001953125 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚27371.18115234375”¼@‘£‚(:† ¯úõÒÉð§Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.311' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.437 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»@fÎ]¤‚(:Ò ¾úõÒíÌË disk.nvme0n1.in‚ 0.00390625& disk.nvme1n1.out‚18187.91259765625 disk.dm-1.out‚7703.068359375 disk.nvme0c0n1.in‚ 0.00390625# memory_percent‚96.9984769821167( proc.memory.percent‚18.4262752532959% disk./.usageGB‚1166.8614120483398 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0( proc.memory.availableMB‚ 491.76953125 proc.cpu.threads‚17 network.recv‚ 1.322556e+06' disk./.usagePercent‚76.004651493134 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 155.34375 disk.dm-1.in‚0' disk.nvme0c0n1.out‚27374.5322265625% disk.nvme0n1.out‚27374.5322265625( disk.nvme1c1n1.out‚18187.91259765625 disk.rbd0.in‚0" proc.memory.rssMB‚ 3018.9609375 cpu‚25.636596971330817# disk.dm-0.out‚37527.71142578125 network.sent‚ 8.697431e+06ÓMÔž¥‚(:“ ¾úõÒ¡ˆ•Ì gpu.0.gpu‚62.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚288.109' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.03633333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0І)w2¦- _wandbruntime‚4981  _runtime‚4981ÁÉM'ǧÁ % _timestamp‚1.7845118078981345e+09 _runtime‚4981.473836062 _step‚453 & lr-AdamW/pg1‚0.0005156579547272935 & lr-AdamW/pg2‚0.0005156579547272935 trainer/global_step‚5549Ý$ Ì¨Æ & lr-AdamW/pg2‚0.0005156579547272935 trainer/global_step‚5549 % _timestamp‚1.7845118078981345e+09 _runtime‚4981.473836062 _step‚453 & lr-AdamW/pg1‚0.0005156579547272935Å5*¤Ì2©- _wandbruntime‚4981  _runtime‚4981§Þgwç'ªáO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚849675404806117 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚843252095711088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99704610 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚848209549368832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6433383259136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6469861825024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚848209549368832 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6433383259136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9682821 % _timestamp‚1.7845118079011316e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6423666695663 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22947426 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9682824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13264905 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚843252078671728 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22947729 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184572251527680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6423326134389 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚854642574164480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚854642574164480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54863196 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚848209532329472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191042243525632 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184572270139904 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44851557 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6433041835008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚849675762406751‘ïMkì'«æO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚854642574164480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚854642574164480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54863196 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚848209532329472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191042243525632 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184572270139904 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44851557 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6433041835008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚849675762406751 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚849675404806117 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚843252095711088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99704610 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚848209549368832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6433383259136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6469861825024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚848209549368832 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6433383259136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9682821 % _timestamp‚1.7845118079011316e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6423666695663 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22947426 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9682824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13264905 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚843252078671728 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22947729 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184572251527680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6423326134389 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303ÆÙ¯¯%2¬- _wandbruntime‚4982  _runtime‚4982={rJ·­ëM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚5549 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚854792317236224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184605431797760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 _step‚455 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54862709 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99732218 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9684558 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44859352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9684555 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22951423 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚843399509339200 _runtime‚4982.087315485 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22951718 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚843399526378560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44859350 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6434151933952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99722059 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13267160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44859352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚848357823870976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 P8DeviceStatsMoni' -t:tor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54872866 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚848357823870976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191076410634752 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845118085114818e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚854792317236224 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44859352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚848357823870976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54872866 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚848357823870976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191076410634752 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845118085114818e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚854792317236224 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99732218 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9684558 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44859352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9684555 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22951423 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚843399509339200 _runtime‚4982.087315485 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22951718 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚843399526378560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44859350 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6434151933952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99722059 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13267160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13379447 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6525782301184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23148968 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45241209 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192757065945088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚862111868373504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚850623629084688 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6479149909071 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186231265031680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45241207 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚5599 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45241207 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6488952360960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6478809347797 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚855623257436672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55337705 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚855623257436672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186231283643904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚5025.231055143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100589057 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9769524 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚857102778993759 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚862112226836992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6488610936832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚855623274476032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚850623612045328 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚857102421393125 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚862112226836992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 $ _timestamp‚1.784511851651953e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚862111868373504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55347848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13379736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55347848 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6488610936832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23149260 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6525670740480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55337705 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚458 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192757065945088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚862111868373504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_s†ö§plit_bytes.large_pool.current‚18612224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚850623629084688 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6479149909071 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186231265031680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45241207 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚5599 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45241207 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6488952360960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6478809347797 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚855623257436672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55337705 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚855623257436672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186231283643904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚5025.231055143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100589057 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9769524 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚857102778993759 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚862112226836992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6488610936832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚855623274476032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚850623612045328 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚857102421393125 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚862112226836992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 $ _timestamp‚1.784511851651953e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚862111868373504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55347848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13379736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55347848 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6488610936832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23149260 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6525670740480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55337705 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚458 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13379447 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6525782301184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ÊŒ ]¾2¾- _wandbruntime‚5025  _runtime‚5025™pÜËõ&¿ïM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6489721035776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13381735 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚855771531938816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192791233005056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚459 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55357518 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6526897159680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚192791361073664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6480258297435 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55357518 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13382017 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100596361 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100606520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100596361 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚862261611445248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚855771548978176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45249002 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9771258 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186264445301760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚862261611445248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚862261252974592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚855771548978176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6490062467072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45249000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23153275 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚850771059752160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23152990 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6526787703296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6489721035776 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45249002 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7845118522766132e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚855771531938816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159nµ“¦ú&ÀôM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚459 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55357518 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6526897159680 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚192791361073664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6480258297435 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55357518 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13382017 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100596361 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100606520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100596361 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚862261611445248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚855771548978176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45249002 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9771258 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186264445301760 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚862261611445248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚862261252974592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚855771548978176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6490062467072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45249000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23153275 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚850771059752160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23152990 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6526787703296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6489721035776 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45249002 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7845118522766132e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚855771531938816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6489721035776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13381735 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚855771531938816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚192791233005056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0˘¹ x2Á- _wandbruntime‚5025  _runtime‚5025,Ⳡƒ /loss/FRt1/assignment_loss‚1.604953670490056 0loss/FRt2/assignment_loss‚0.8145213099541487 2loss/FBt1/assignment_loss‚0.014089674962217006 /loss/FBt2/detection_loss‚0.8059202992590144 _step‚460 2loss/SRqqt1/assignment_loss‚0.5483295277225638 /loss/FRt1/detection_loss‚0.5165311966248983 /loss/FRt2/detection_loss‚1.5866395916802705 1loss/SRqqt2/detection_loss‚0.6722437861213908 %loss/total_loss‚8.230318969932261 trainer/global_step‚5599 % _timestamp‚1.7845118522781746e+09 _runtime‚5025.853562708 3loss/SRqqt2/assignment_loss‚0.48372110645067623 2loss/FBt2/assignment_loss‚0.002039693688444239 1loss/SRqqt1/detection_loss‚0.6621457149309141 .loss/FBt1/detection_loss‚0.519183398047666 epoch‚13óµ|È /loss/FRt1/detection_loss‚0.5165311966248983 /loss/FRt2/detection_loss‚1.5866395916802705 1loss/SRqqt2/detection_loss‚0.6722437861213908 %loss/total_loss‚8.230318969932261 trainer/global_step‚5599 % _timestamp‚1.7845118522781746e+09 _runtime‚5025.853562708 3loss/SRqqt2/assignment_loss‚0.48372110645067623 2loss/FBt2/assignment_loss‚0.002039693688444239 1loss/SRqqt1/detection_loss‚0.6621457149309141 .loss/FBt1/detection_loss‚0.519183398047666 epoch‚13 /loss/FRt1/assignment_loss‚1.604953670490056 0loss/FRt2/assignment_loss‚0.8145213099541487 2loss/FBt1/assignment_loss‚0.014089674962217006 /loss/FBt2/detection_loss‚0.8059202992590144 _step‚460 2loss/SRqqt1/assignment_loss‚0.5483295277225638Ì1uóbÄ‚(:× úúõÒ‚¶ Ì) disk./.usagePercent‚76.00465372940012% disk./.usageGB‚1166.8614463806152 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚37533.51416015625 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚27380.8291015625 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚27380.8291015625( disk.nvme1c1n1.out‚18190.30517578125 disk.md0.in‚0' proc.memory.availableMB‚ 495.6015625 network.recv‚ 1.337584e+06 disk.md0.out‚ 156.515625$ memory_percent‚96.97508811950684 cpu‚25.64268229995567 proc.cpu.threads‚17 disk.dm-1.out‚7703.427734375 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme1n1.out‚18190.30517578125# proc.memory.rssMB‚ 3013.34765625* proc.memory.percent‚18.392014503479004 network.sent‚ 8.784232e+06ã×5ŸÅ‚(:” úúõÒ¬íæÌ gpu.0.gpu‚58.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚118.368' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.455999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é ½2Æ- _wandbruntime‚5042  _runtime‚5042§æuæoÇåz 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9929608106613159 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2966010570526123 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.763136088848114 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912348985671997 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12100891023874283 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6628232598304749 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.027208050712943077 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2932523190975189 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7037004828453064 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.669493556022644 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6666538715362549 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚¦b©|ü60.6008073091506958 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.8144410252571106 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984719157218933 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6357856392860413 +jet/accuracy_1_of_1‚0.33751609921455383 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.515625 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3818420171737671 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985670447349548 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48668989539146423 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9094671607017517 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +particle/sensitivity‚0.6297940015792847 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21261145174503326 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23394906520843506 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7595534920692444 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18693357706069946 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.864791750907898 trainer/global_step‚5599 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979004859924316 particle/accuracy_1_of_1‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22293996810913086 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9899949431419373 particle/accuracy_1_of_3‚1 +jet/accuracy_3_of_4‚0.09281877428293228 particle/accuracy_1_of_2‚1 jet/accuracy_5_of_5‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23394906520843506 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.030824681743979454 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2475895881652832 )jet/accuracy_1_of_3‚0.935248076915741 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21666666865348816 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5820833444595337 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951820373535156 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9799898266792297 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.864791750907898 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6333220601081848 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8311792016029358 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9223443269729614 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_2_of_4‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5010531544685364 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4915257394313812 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13742955029010773 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3883267939090729 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.027208050712943077 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_4‚0.9722879528999329 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 epoch‚13 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965580701828003 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9866749048233032 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6765152812004089 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6829938292503357 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.047367725521326065 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7648875713348389 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22499804198741913 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.013766873627901077 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 _step‚461 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19863244891166687 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.689611554145813 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985670447349548 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6922546625137329 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20763301849365234 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873634576797485 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4575778841972351 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8934934735298157 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8362353444099426 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24623382091522217 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6614733338356018 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8967120051383972 7validation_average_jet_accuracy‚0.38067296147346497 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14777334034442902 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.030747251585125923 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19397518038749695 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6943589448928833 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7109375596046448 jet/accuracy_4_of_5‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.713252067565918 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.024917028844356537 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_2_of_2‚0.15017765760421753 )loss/val_total_loss‚7.895280845750293 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.811947762966156 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7513477802276611 /particle/accuracy_4_of_4‚0.9535135626792908 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7482360601425171 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18782353401184082 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5955989360809326 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23523777723312378 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _runtime‚5042.022513856 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7856313586235046 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7285518050193787 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9562684297561646 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7007753849029541 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025713834911584854 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958211779594421 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21265943348407745 +jet/accuracy_3_of_3‚0.07736726850271225 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29503679275512695 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6635388135910034 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7592657208442688 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912253022193909 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949805736541748 (particle/accuracy‚0.8491520881652832 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6502135992050171 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7549707293510437 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7795966267585754 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8995843529701233 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13066650927066803 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03051590360701084 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2125178575515747 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.9094671607017517 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03266396000981331 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3283778429031372 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.811246395111084 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21875974535942078 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7639775276184082 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4838721752166748 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7801668047904968 *jet/accuracy_2_of_3‚0.5528932213783264 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.903656005859375 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7157681584358215 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24936392903327942 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6458050012588501 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 *jet/accuracy_2_of_4‚0.8017749786376953 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7070561051368713 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959833025932312 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6628232598304749 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988323450088501 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19924375414848328 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 +particle/specificity‚0.9091439843177795 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7046815752983093 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17675621807575226 'particle/f_score‚0.6419728398323059 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04386220499873161 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24741055071353912 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.031184082850813866 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2688281536102295 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 % _timestamp‚1.7845118684463806e+09 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8832578063011169 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38958659768104553 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4904608130455017 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7655159831047058 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456q¿pþp=Èêz =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.763136088848114 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912348985671997 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12100891023874283 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6628232598304749 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.027208050712943077 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2932523190975189 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7037004828453064 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.669493556022644 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6666538715362549 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6008073091506958 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.8144410252571106 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984719157218933 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6357856392860413 +jet/accuracy_1_of_1‚0.33751609921455383 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.515625 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3818420171737671 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985670447349548 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48668989539146423 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9094671607017517 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +particle/sensitivity‚0.6297940015792847 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21261145174503326 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23394906520843506 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7595534920692444 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18693357706069946 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.864791750907898 trainer/global_step‚5599 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979004859924316 particle/accuracy_1_of_1‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22293996810913086 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9899949431419373 particle/accuracy_1_of_3‚1 +jet/accuracy_3_of_4‚0.09281877428293228 particle/accuracy_1_of_2‚1 jet/accuracy_5_of_5‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23394906520843506 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.030824681743979454 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2475895881652832 )jet/accuracy_1_of_3‚0.935248076915741 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21666666865348816 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5820833444595337 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951820373535156 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9799898266792297 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.864791750907898 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6333220601081848 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8311792016029358 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9223443269729614 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_2_of_4‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5010531544685364 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4915257394313812 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13742955029010773 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3883267939090729 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.027208050712943077 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_4‚0.9722879528999329 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 epoch‚13 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965580701828003 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9866749048233032 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6765152812004089 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6829938292503357 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.047367725521326065 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7648875713348389 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22499804198741913 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.013766873627901077 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 _step‚461 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19863244891166687 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.689611554145813 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985670447349548 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6922546625137329 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20763301849365234 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873634576797485 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4575778841972351 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8934934735298157 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8362353444099426 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24623382091522217 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6614733338356018 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8967120051383972 7validation_average_jet_accuracy‚0.38067296147346497 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14777334034442902 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.030747251585125923 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19397518038749695 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6943589448928833 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7109375596046448 jet/accuracy_4_of_5‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.713252067565918 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.024917028844356537 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_2_of_2‚0.15017765760421753 )loss/val_total_loss‚7.895280845750293 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.811947762966156 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7513477802276611 /particle/accuracy_4_of_4‚0.9535135626792908 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7482360601425171 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18782353401184082 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5955989360809326 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23523777723312378 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _runtime‚5042.022513856 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7856313586235046 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7285518050193787 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9562684297561646 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7007753849029541 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025713834911584854 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958211779594421 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21265943348407745 +jet/accuracy_3_of_3‚0.07736726850271225 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29503679275512695 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6635388135910034 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7592657208442688 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912253022193909 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949805736541748 (particle/accuracy‚0.8491520881652832 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6502135992050171 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7549707293510437 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7795966267585754 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8995843529701233 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13066650927066803 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03051590360701084 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2125178575515747 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.9094671607017517 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03266396000981331 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3283778429031372 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.811246395111084 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21875974535942078 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7639775276184082 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4838721752166748 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7801668047904968 *jet/accuracy_2_of_3‚0.5528932213783264 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.903656005859375 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7157681584358215 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24936392903327942 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6458050012588501 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 *jet/accuracy_2_of_4‚0.8017749786376953 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7070561051368713 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959833025932312 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6628232598304749 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988323450088501 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19924375414848328 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 +particle/specificity‚0.9091439843177795 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7046815752983093 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17675621807575226 'particle/f_score‚0.6419728398323059 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04386220499873161 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24741055071353912 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.031184082850813866 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2688281536102295 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 % _timestamp‚1.7845118684463806e+09 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8832578063011169 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38958659768104553 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4904608130455017 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7655159831047058 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9929608106613159 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2966010570526123 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0ÍßgÓ"`É‚(:Õ ‰ûõÒ´ïÒË disk.md0.in‚0 disk.md0.out‚ 156.78515625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625' proc.memory.availableMB‚ 738.1328125 network.sent‚ 8.846734e+06 disk.nvme1n1.in‚0# disk.nvme1n1.out‚18436.40234375# disk.dm-0.out‚37534.57666015625% disk.nvme0n1.out‚27381.8291015625 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚18436.40234375 cpu‚25.642661075559175' disk.nvme0c0n1.out‚27381.8291015625 disk.nvme0n1.in‚ 0.00390625$ memory_percent‚95.49479484558105* proc.memory.percent‚18.392300605773926 proc.cpu.threads‚17 disk.rbd0.out‚0 network.recv‚ 1.34264e+06 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0" disk.dm-1.out‚7948.87646484375# proc.memory.rssMB‚ 3013.39453125) disk./.usagePercent‚76.00471957501324% disk./.usageGB‚1166.8624572753906×ùŸÊ‚(:” ‰ûõÒ®³Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚53.0 gpu.0.powerWatts‚119.071' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.690333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fFë9ZË‚(:Ï ˜ûõÒ–åÃË* proc.memory.percent‚18.392300605773926 proc.cpu.threads‚17 network.sent‚ 8.892656e+06) disk./.usagePercent‚76.00472901702567 disk.dm-1.in‚0" disk.nvme0n1.out‚ 27384.0078125 disk.md0.in‚0 disk.md0.out‚ 157.2421875 cpu‚25.64219244729345% disk./.usageGB‚1166.8626022338867" disk.dm-1.out‚7996.93896484375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme0c0n1.out‚ 27384.0078125 disk.dm-0.in‚ 0.00390625$ memory_percent‚95.49503326416016% proc.memory.availableMB‚ 738.09375 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚18485.1689453125 disk.nvme1n1.in‚0% disk.nvme1n1.out‚18485.1689453125# proc.memory.rssMB‚ 3013.39453125 network.recv‚ 1.34769e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625# disk.dm-0.out‚37536.48095703125¤3;ڞ̂(:“ ˜ûõÒœüýË gpu.0.gpu‚57.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.676' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.55866666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿNG‚lÍ‚(:× §ûõÒ´õ–Ë network.recv‚ 1.351203e+06% disk./.usageGB‚1166.8614501953125 disk.nvme1n1.in‚0 disk.md0.out‚ 157.828125 disk.dm-1.in‚0 disk.dm-1.out‚8194.439453125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 network.sent‚ 8.895679e+06 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚27385.1318359375# dis -Ýçök.dm-0.out‚37537.11376953125' disk.nvme0c0n1.out‚27385.1318359375# memory_percent‚95.4921007156372( proc.memory.availableMB‚ 738.57421875 cpu‚25.64158200739833) disk./.usagePercent‚76.00465397787413 disk.nvme1c1n1.in‚0 proc.cpu.threads‚17( disk.nvme1c1n1.out‚18683.40283203125& disk.nvme1n1.out‚18683.40283203125 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0# proc.memory.rssMB‚ 3013.39453125* proc.memory.percent‚18.392300605773926ÉÕ¢žÎ‚(:“ §ûõÒµ‹ëË gpu.0.gpu‚91.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚275.641' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.88033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÆ¡A2Ï- _wandbruntime‚5086  _runtime‚5086Α&Åп & lr-AdamW/pg1‚0.0005120128905436099 & lr-AdamW/pg2‚0.0005120128905436099 trainer/global_step‚5649 $ _timestamp‚1.784511912619227e+09 _runtime‚ 5086.19604049 _step‚462¸±òÊÑÄ _runtime‚ 5086.19604049 _step‚462 & lr-AdamW/pg1‚0.0005120128905436099 & lr-AdamW/pg2‚0.0005120128905436099 trainer/global_step‚5649 $ _timestamp‚1.784511912619227e+09Î`îR§2Ò- _wandbruntime‚5086  _runtime‚5086ÀHGµé'ÓãO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚859916418466512 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚864979343458816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9860337 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6554936971381 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 % _timestamp‚1.7845119126199558e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101667666 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚194515218157056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6602241271808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23393059 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194515098469888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚864979326419456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55913999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45753667 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6564839301120 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6564497876992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55903856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚864979343458816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚864979326419456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚871543824296448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9860340 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45753665 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6554596410107 _runtime‚ 5086.1961267 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚871544182759936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚871543824296448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13532719 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55903856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101657521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚866471355437893 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚5649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13532430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187912857198080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚866470997837259|°äÒî'ÔèO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6564839301120 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187912875810304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚859916401427152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45753667 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23392767 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6564497876992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55903856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚864979343458816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚864979326419456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚871543824296448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9860340 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45753665 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6554596410107 _runtime‚ 5086.1961267 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚871544182759936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚871543824296448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13532719 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55903856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101657521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚866471355437893 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚5649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13532430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187912857198080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚866470997837259 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚859916418466512 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚864979343458816 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9860337 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6554936971381 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 % _timestamp‚1.7845119126199558e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101667666 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚194515218157056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6602241271808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23393059 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194515098469888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚864979326419456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55913999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45753667 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6564839301120 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6565607975936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55913512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6565949407232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6556045359745 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45761458 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194549393611776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23396788 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45761458 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6603457211392 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚866619894493729 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚866619536892983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101685129 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187946056080384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 % _timestamp‚1.7845119132410407e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚871693567368192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚860063832094624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6565607975936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚5086.816879647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9862071 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚865127600921600 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚871693567368192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚871693208897536 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55913512 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 233435688962Ñ_.×ôM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55913512 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6565949407232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6556045359745 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45761458 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194549393611776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23396788 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45761458 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6603457211392 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚866619894493729 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚866619536892983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101685129 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187946056080384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 % _timestamp‚1.7845119132410407e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚871693567368192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚860063832094624 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6565607975936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚5086.816879647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9862071 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚865127600921600 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚871693567368192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚871693208897536 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55913512 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚865127617960960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101674970 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13534987 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚860063849133984 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6565607975936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180ÐÛtÃ2Ø- _wandbruntime‚5086  _runtime‚5086$JÙ 0loss/FRt1/assignment_loss‚1.7477577795465309 1loss/SRqqt2/detection_loss‚0.7081282714992261 _runtime‚5086.818109409 0loss/FRt2/assignment_loss‚0.7547607496556943 2loss/SRqqt2/assignment_loss‚0.5450016250605578 1loss/FBt2/assignment_loss‚0.03516720781242481 .loss/FRt2/detection_loss‚1.516798157619211 /loss/FBt2/detection_loss‚0.7700246888478952 epoch‚14 $ _timestamp‚1.784511913242662e+09 2loss/SRqqt1/assignment_loss‚0.5137070938701598 2loss/FBt1/assignment_loss‚0.009523941365865814 $loss/total_loss‚8.21864070102741 _step‚465 /loss/FRt1/detection_loss‚0.5039603956005196 1loss/SRqqt1/detection_loss‚0.7320927573360039 0loss/FBt1/detection_loss‚0.38171803281332056 trainer/global_step‚5649¹àí Ú† 0loss/FRt1/assignment_loss‚1.7477577795465309 1loss/SRqqt2/detection_loss‚0.7081282714992261 _runtime‚5086.818109409 0loss/FRt2/assignment_loss‚0.7547607496556943 2loss/SRqqt2/assignment_loss‚0.5450016250605578 1loss/FBt2/assignment_loss‚0.03516720781242481 .loss/FRt2/detection_loss‚1.516798157619211 /loss/FBt2/detection_loss‚0.7700246888478952 epoch‚14 $ _timestamp‚1.784511913242662e+09 2loss/SRqqt1/assignment_loss‚0.5137070938701598 2loss/FBt1/assignment_loss‚0.009523941365865814 $loss/total_loss‚8.21864070102741 _step‚465 /loss/FRt1/detection_loss‚0.5039603956005196 1loss/SRqqt1/detection_loss‚0.7320927573360039 0loss/FBt1/detection_loss‚0.38171803281332056 trainer/global_step‚5649ÑtÅË­bÛ‚(:× ¶ûõÒÒõËË' disk.nvme1c1n1.out‚18684.2509765625 disk.dm-1.out‚8194.564453125 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.md0.out‚ 158.2265625 disk.dm-0.in‚ 0.00390625 proc.cpu.threads‚17( disk./.usagePercent‚76.0046577049843 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚18684.2509765625* proc.memory.percent‚18.392300605773926 cpu‚25.641015959823914 network.sent‚ 8.914417e+06 network.recv‚ 1.35444e+06% disk./.usageGB‚1166.8615074157715 disk.rbd0.out‚0( disk.nvme0c0n1.out‚27387.14501953125& disk.nvme0n1.out‚27387.14501953125 disk.md0.in‚0# disk.dm-0.out‚37538.99755859375$ memory_percent‚95.49250602722168' proc.memory.availableMB‚ 738.5078125# proc.memory.rssMB‚ 3013.39453125s~Ü‚(:’ ¶ûõÒ力Ì gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚263.33' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.77666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ hweÝ‚(:Ú ÅûõÒä¥ÏË% disk./.usageGB‚1166.8614883422852' disk.nvme0c0n1.out‚27388.0537109375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0" disk.dm-1.out‚8194.69677734375 network.sent‚ 8.964768e+06 network.recv‚ 1.359764e+06 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚18684.8701171875 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚37539.60693359375$ memory_percent‚95.49250602722168% disk.nvme0n1.out‚27388.0537109375 disk.md0.out‚ 158.58203125 disk.rbd0.in‚0* proc.memory.percent‚18.392300605773926 cpu‚25.640554719064227) disk./.usagePercent‚76.00465646261424% disk.nvme1n1.out‚18684.8701171875 disk.dm-1.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 738.5078125# proc.memory.rssMB‚ 3013.39453125 proc.cpu.threads‚17¶wõ³žÞ‚(:“ ÅûõÒທÌ gpu.0.gpu‚93.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚209.432' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚69.81066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$™ 2ß- _wandbruntime‚5129  _runtime‚51298­XÇàÁ _runtime‚5129.854354075 _step‚466 & lr-AdamW/pg1‚0.0005101705718934831 & lr-AdamW/pg2‚0.0005101705718934831 trainer/global_step‚5699 % _timestamp‚1.7845119562788186e+09R PrÌáÆ _runtime‚5129.854354075 _step‚466 & lr-AdamW/pg1‚0.0005101705718934831 & lr-AdamW/pg2‚0.0005101705718934831 trainer/global_step‚5699 % _timestamp‚1.7845119562788186e+09Ò`ð2â- _wandbruntime‚5129  _runtime‚5129F×”qê'ãäO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46143317 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9947040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚867287934800752 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23595180 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879013476968960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13648422 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6620066978816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚879013476968960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102541968 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879013118505472 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚872393051526656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6620408402944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0*Í ºª!äéO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚879013476968960 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102541968 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879013118505472 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚872393051526656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6620408402944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46143317 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9947040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚867287934800752 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23595180 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879013476968960 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13648422 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6620066978816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344Óå,aå‚(:Ö ÔûõÒ•’¶Ë& proc.memory.availableMB‚ 738.234375 network.sent‚ 8.96769e+06) disk./.usagePercent‚76.00462937894696 disk.md0.out‚ 158.9921875 disk.dm-1.in‚0" disk.dm-1.out‚8194.73974609375 proc.cpu.threads‚17 network.recv‚ 1.363196e+06% disk./.usageGB‚1166.8610725402832 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme1n1.out‚18685.638671875 disk.md0.in‚0" disk.dm-0.out‚37541.5068359375 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚27390.10888671875& disk.nvme1c1n1.out‚18685.638671875 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625$ memory_percent‚95.49417495727539# proc.memory.rssMB‚ 3013.39453125* proc.memory.percent‚18.392300605773926 cpu‚25.639952793978093( disk.nvme0c0n1.out‚27390.10888671875 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0*tünœæ‚(:‘ ÔûõÒñÔ¯Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚169.532' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚56.51066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sqúÌ2ç- _wandbruntime‚5130  _runtime‚5130¸Abîõ&èïM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46151110 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23599428 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚5699 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9948771 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚872541326028800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102549272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13650357 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56408321 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46151110 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56398164 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6659234729472 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13650654 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚297 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚872541343068160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879162861577216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚874046553479991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚867435365468224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚468 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚872541343068160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879162861577216 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6621518509056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚196264407478784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚872541326028800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102559431 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6659337894400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚874046911080737 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56398164 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6621177077760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚867435382507584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102559431 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6621177077760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879162503106560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46151108 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189605050972160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845119569028158e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23599128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9948774 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6611528573153 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56408321 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚196264285701632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5130.478584001 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102549272 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6611188011767°ÑËXú&éôM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46151108 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56408321 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46151110 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56398164 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6659234729472 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13650654 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚297 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚872541343068160 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879162861577216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚874046553479991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚867435365468224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚468 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚872541343068160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚879162861577216 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6621518509056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚196264407478784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚872541326028800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102559431 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6659337894400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚874046911080737 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56398164 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6621177077760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚867435382507584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102559431 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6621177077760 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879162503106560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46151108 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189605050972160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845119569028158e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23599128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9948774 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6611528573153 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56408321 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚196264285701632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5130.478584001 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102549272 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6611188011767 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46151110 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23599428 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚5699 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9948771 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚872541326028800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102549272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13650357 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6665562836923 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197944882442240 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6665903398197 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46532965 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6675977504768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56883303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10033740 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13762776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6675636080640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56873160 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109201408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚881325388611909 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103416270 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚879806776633856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191230902818304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚886482771177984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6675977504768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6675636080640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚879806776633856 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13762505 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197944991643648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚886482412714496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚886482412714496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _runtime‚5173.056235513 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647ÇÍSð'öêO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚874659468174352 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6665562836923 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197944882442240 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6665903398197 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46532965 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6675977504768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56883303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10033740 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13762776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6675636080640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56873160 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109201408 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚881325388611909 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103416270 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚879806776633856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191230902818304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚886482771177984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6675977504768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6675636080640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚879806776633856 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13762505 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197944991643648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚886482412714496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚886482412714496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _runtime‚5173.056235513 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚886482771177984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56873160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6714088825344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 $ _timestamp‚1.784511999480774e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚881325031011275 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚874659485213712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚5749 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚879806793673216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46532967 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23796516 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191230884206080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6677087610880 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6676746179584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚879955051136000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6715205961216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6715115379200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13765033 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191264064476160 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _step‚472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚874806898841824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6667011786561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6666671225175 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚5173.679972915 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46540760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23800226 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197979289049600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103423574 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103433733 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56882816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56892973 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0öHÅ-¹ùõM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚881473570066999 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109194240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56892973 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6677087610880 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6676746179584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚879955051136000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6715205961216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6715115379200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13765033 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191264064476160 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _step‚472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚874806898841824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6667011786561 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6666671225175 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚5173.679972915 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46540760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23800226 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197979289049600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceSta2ÜiB tsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103423574 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103433733 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56882816 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56892973 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46922617 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23998289 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6721386611605 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 $ _timestamp‚1.784512042864264e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 _step‚475 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23998563 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192889916322304 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13877852 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚5799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚199659848237568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57357812 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚893952065387008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6721046050331 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6769931915264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104290572 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚887220501741056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6731205182464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6731205182464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚893951706923520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚882031018587312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚888752047598283 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚887220518780416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192889897710080 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57367955 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚887220518780416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57367955 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6769828743168 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6731546606592 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚199659726453248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6731546606592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚893952065387008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46922617 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚893951706923520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10120440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10120437 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13878123 _runtime‚5216.439906434 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46922615 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46922615I)æŠñ'ˆëO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚887220518780416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192889897710080 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57367955 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚887220518780416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57367955 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6769828743168 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6731546606592 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚199659726453248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6731546606592 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚893952065387008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46922617 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚893951706923520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10120440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10120437 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13878123 _runtime‚5216.439906434 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46922615 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46922615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57357812 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46922617 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23998289 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6721386611605 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 $ _timestamp‚1.784512042864264e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 _step‚475 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23998563 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192889916322304 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13877852 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚5799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚199659848237568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57357812 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚893952065387008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6721046050331 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6769931915264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104290572 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚887220501741056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6731205182464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6731205182464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚893951706923520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚882031018587312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚888752047598283 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Û˜²2‰- _wandbruntime‚5217  _runtime‚5217(Üþñ ŠñM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚882178449254784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6732656712704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚887368776243200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10122171 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57377625 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46930408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7845120434839816e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚199694022832128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6770944851968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13880432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚887368776243200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10122174 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚199694142512128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6722494999969 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚887368793282560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6732315281408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46930410 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _step‚476 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚888900944254753 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6732315281408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104297876 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24002329 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚888900586654007 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚882178432215424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚5799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104308035 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚894101449995264 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104308035 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57367468 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0FPÝ@ü&‹öM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6770944851968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13880432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚887368776243200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10122174 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚199694142512128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6722494999969 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚887368793282560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6732315281408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46930410 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _step‚476 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚888900944254753 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6732315281408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104297876 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24002329 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚888900586654007 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚882178432215424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚5799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104308035 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚894101449995264 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104308035 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57367468 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚882178449254784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6732656712704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚887368776243200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10122171 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57377625 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46930408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7845120434839816e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚199694022832128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344ÜŸ»m2Œ- _wandbruntime‚5217  _runtime‚5217+Hyk ƒ 2loss/SRqqt2/assignment_loss‚0.5453364304575785 1loss/SRqqt1/detection_loss‚0.6892300443574616 %loss/total_loss‚7.780852766415294 trainer/global_step‚5799 /loss/FRt2/detection_loss‚1.5428707222982898 1loss/SRqqt2/detection_loss‚0.6819784966392035 epoch‚14 _step‚477 0loss/FRt2/assignment_loss‚0.7382981916844557 2loss/SRqqt1/assignment_loss‚0.4707424932123657 3loss/FBt2/assignment_loss‚0.0003569087204248129 /loss/FRt1/detection_loss‚0.5165466203981982 /loss/FBt2/detection_loss‚0.3987071845046383 _runtime‚5217.060301652 /loss/FRt1/assignment_loss‚1.697853644057561 1loss/FBt1/assignment_loss‚0.00037222978975958 /loss/FBt1/detection_loss‚0.4985598002953577 % _timestamp‚1.7845120434850278e+09u48Žˆ /loss/FRt2/detection_loss‚1.5428707222982898 1loss/SRqqt2/detection_loss‚0.6819784966392035 epoch‚14 _step‚477 0loss/FRt2/assignment_loss‚0.7382981916844557 2loss/SRqqt1/assignment_loss‚0.4707424932123657 3loss/FBt2/assignment_loss‚0.0003569087204248129 /loss/FRt1/detection_loss‚0.5165466203981982 /loss/FBt2/detection_loss‚0.3987071845046383 _runtime‚5217.060301652 /loss/FRt1/assignment_loss‚1.697853644057561 1loss/FBt1/assignment_loss‚0.00037222978975958 /loss/FBt1/detection_loss‚0.4985598002953577 % _timestamp‚1.7845120434850278e+09 2loss/SRqqt2/assignment_loss‚0.5453364304575785 1loss/SRqqt1/detection_loss‚0.6892300443574616 %loss/total_loss‚7.780852766415294 trainer/global_step‚5799ݧÚ}[a‚(:Ö ®üõÒ°½¦Ë$ disk./.usageGB‚1163.906078338623 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0c0n1.out‚27403.798828125" proc.memory.rssMB‚ 3013.3984375 cpu‚25.63683239119336! disk.dm-1.out‚8195.3544921875 network.sent‚ 9.148613e+06( disk.nvme1c1n1.out‚18690.10693359375 disk.md0.out‚ 161.2265625 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚37554.28076171875 disk.dm-1.in‚0 disk.rbd0.in‚0 network.recv‚ 1.388906e+06 disk.nvme1n1.in‚0& disk.nvme1n1.out‚18690.10693359375 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625$ disk.nvme0n1.out‚27403.798828125$ memory_percent‚95.49181461334229) disk./.usagePercent‚75.81215296131725 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 738.62109375* proc.memory.percent‚18.392324447631836 proc.cpu.threads‚17gÈ™ˆœ‚(:‘ ®üõÒøÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.297' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.76566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ðva‘‚(:Ö ½üõÒ°‘¶Ë network.recv‚ 1.394209e+06) disk./.usagePercent‚75.81215320979126 disk.nvme1n1.in‚0% disk.nvme1n1.out‚18690.6767578125# disk.dm-0.out‚37554.98779296875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625 network.sent‚ 9.212854e+06 disk.md0.in‚0 disk.rbd0.out‚0' disk.nvme1c1n1.out‚18690.6767578125( proc.memory.availableMB‚ 738.62109375 cpu‚25.636249890567864 proc.cpu.threads‚17" disk.dm-1.out‚8195.45556640625 disk.md0.out‚ 161.578125 disk.dm-0.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625$ disk.nvme0n1.out‚27404.802734375 disk.nvme1c1n1.in‚0$ memory_percent‚95.49181461334229" proc.memory.rssMB‚ 3013.3984375% disk./.usageGB‚1163.9060821533203& disk.nvme0c0n1.out‚27404.802734375* proc.memory.percent‚18.392324447631836 disk.rbd0.in‚0ÀÅÌž’‚(:“ ½üõÒ°àòË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.875' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.29166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+saTg“‚(:Ü ÌüõÒ£éÇË( proc.memory.availableMB‚ 738.62109375 network.sent‚ 9.215818e+06( disk./.usagePercent‚75.9920409400889 disk.dm-1.in‚0( disk.nvme0c0n1.out‚28815.20556640625( disk.nvme1c1n1.out‚18692.69677734375* proc.memory.percent‚18.392324447631836 cpu‚25.63586033451878 proc.cpu.threads‚17 network.recv‚ 1.397177e+06# disk.dm-0.out‚38960.29638671875& disk.nvme0n1.out‚28815.20556640625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0& disk.nvme1n1.out‚18692.69677734375$ memory_percent‚95.49181461334229 disk.nvme0n1.in‚ 0.00390625% disk./.usageGB‚1166.6678085327148" disk.dm-1.out‚8195.51416015625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.md0.in‚0 disk.md0.out‚ 162.01171875" proc.memory.rssMB‚ 3013.3984375 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0clPŸœ”‚(:‘ ÌüõÒËŽœÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚281.714' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.90466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à3A2•- _wandbruntime‚5258  _runtime‚5258dßÄÆ–À $ _timestamp‚1.784512085413351e+09 _runtime‚5258.988941057 _step‚478 & lr-AdamW/pg1‚0.0005045658921426806 & lr-AdamW/pg2‚0.0005045658921426806 trainer/global_step‚5849ñÕ 6Ë—Å & lr-AdamW/pg2‚0.0005045658921426806 trainer/global_step‚5849 $ _timestamp‚1.784512085413351e+09 _runtime‚5258.988941057 _step‚478 & lr-AdamW/pg1‚0.0005045658921426806ÞbÇB?2˜- _wandbruntime‚5258  _runtime‚5258×¢jì'™æO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57852607 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57842464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6776869825013 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47312267 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47312265 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6786774284288 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6787115708416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚901421359596032 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚901421001132544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47312265 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105154729 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201374707806208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194548929826304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚889402534921552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 $ _timestamp‚1.784512085414319e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47312267 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13993189 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6776529263739 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24200053 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚896179421785925 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13992916 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚894634243887616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194548911214080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201374586021888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6825777979904 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _step‚479 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57842464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10207140 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10207137 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _runtime‚5258.989882109 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6825674807808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚894634226848256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚894634226848256 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6787115708416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128/¡z^šëO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatÙžÑד'sMonitor.on_train_batch_start/allocation.large_pool.allocated‚47312267 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13993189 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6776529263739 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24200053 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚896179421785925 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13992916 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚894634243887616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194548911214080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201374586021888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6825777979904 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _step‚479 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57842464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10207140 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10207137 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _runtime‚5258.989882109 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6825674807808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚894634226848256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚894634226848256 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6787115708416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57852607 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57842464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6776869825013 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47312267 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47312265 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6786774284288 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6787115708416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚901421359596032 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚901421001132544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47312265 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105154729 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201374707806208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194548929826304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚889402534921552 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 $ _timestamp‚1.784512085414319e+09ßÆ»»g2›- _wandbruntime‚5259  _runtime‚5259™¯Q÷&œñM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47320058 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194582091484160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10208874 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6787884383232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚896327603241015 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚889549965589024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6826787127808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105182337 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57852120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6788225814528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201409004583424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57852120 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚901570744204288 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚5849 _runtime‚5259.914048875 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚894782518389760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6787884383232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24204033 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6777978213377 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182337 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57862277 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6826894487040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚901570744204288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚894782518389760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456Q*Ýéü&öM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚894782518389760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6787884383232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24204033 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6777978213377 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182337 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57862277 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6826894487040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚901570744204288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚894782518389760 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47320058 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194582091484160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10208874 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6787884383232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚896327603241015 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚889549965589024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6826787127808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 105182337 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57852120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6788225814528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201409004583424 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57852120 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚901570744204288 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚5849 _runtime‚5259.914048875 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180àZ"¸2ž- _wandbruntime‚5259  _runtime‚5259K -« Ÿ… 2loss/SRqqt1/assignment_loss‚0.5214255232535371 3loss/FBt2/assignment_loss‚0.0040217767663827024 /loss/FRt2/detection_loss‚1.5814206685496552 1loss/SRqqt1/detection_loss‚0.7232059730294673 /loss/FBt2/detection_loss‚0.4632596306901957 _runtime‚5259.914580796 0loss/FRt2/assignment_loss‚0.8966072826639503 2loss/SRqqt2/assignment_loss‚0.5431321932620348 2loss/FBt1/assignment_loss‚0.004884682413795057 /loss/FRt1/detection_loss‚0.4738498295775278 1loss/SRqqt2/detection_loss‚0.7334408161602012 epoch‚14 trainer/global_step‚5849 0loss/FRt1/assignment_loss‚1.6709219186049171 /loss/FBt1/detection_loss‚0.5114537476040583 %loss/total_loss‚8.127624042575722 % _timestamp‚1.7845120863393104e+09 _step‚481á¤y# Š %loss/total_loss‚8.127624042575722 % _timestamp‚1.7845120863393104e+09 _step‚481 2loss/SRqqt1/assignment_loss‚0.5214255232535371 3loss/FBt2/assignment_loss‚0.0040217767663827024 /loss/FRt2/detection_loss‚1.5814206685496552 1loss/SRqqt1/detection_loss‚0.7232059730294673 /loss/FBt2/detection_loss‚0.4632596306901957 _runtime‚5259.914580796 0loss/FRt2/assignment_loss‚0.8966072826639503 2loss/SRqqt2/assignment_loss‚0.5431321932620348 2loss/FBt1/assignment_loss‚0.004884682413795057 /loss/FRt1/detection_loss‚0.4738498295775278 1loss/SRqqt2/detection_loss‚0.7334408161602012 epoch‚14 trainer/global_step‚5849 0loss/FRt1/assignment_loss‚1.6709219186049171 /loss/FBt1/detection_loss‚0.5114537476040583áøÝØ“Z¡‚(:Ï ÛüõÒúâØË disk.rbd0.out‚0* proc.memory.percent‚18.392324447631836 cpu‚25.635283752938363 proc.cpu.threads‚17 network.sent‚ 9.234601e+06 network.recv‚ 1.40093e+06( disk./.usagePercent‚75.9920578363217$ disk./.usageGB‚1166.668067932129 disk.md0.in‚0# disk.dm-0.out‚39486.10498046875$ memory_percent‚95.49202919006348" proc.memory.rssMB‚ 3013.3984375 disk.nvme0n1.in‚ 0.00390625$ disk.nvme0n1.out‚29335.228515625 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625' proc.memory.availableMB‚ 738.5859375 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚18693.27978515625& disk.nvme1n1.out‚18693.27978515625 disk.dm-1.in‚0& disk.nvme0c0n1.out‚29335.228515625 disk.nvme1n1.in‚0 disk.md0.out‚ 162.390625 disk.dm-1.out‚ 8195.609375 disk.rbd0.in‚0' Ôjb¢‚(:ˆ ÛüõÒ¾Ý—Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.2ùŒcK139273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚121.458' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û’ë{b£‚(:× êüõÒÔúÌË" disk.dm-0.out‚40064.5556640625( disk.nvme0c0n1.out‚29709.19482421875$ memory_percent‚95.49226760864258& proc.memory.availableMB‚ 738.546875" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚18898.36572265625 network.recv‚ 1.405833e+06& disk.nvme1n1.out‚18898.36572265625 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 cpu‚25.634807135886234 proc.cpu.threads‚17 network.sent‚ 9.284941e+06) disk./.usagePercent‚75.99015924639781% disk./.usageGB‚1166.6389198303223 disk.dm-1.out‚ 8195.6796875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚29709.19482421875 disk.md0.in‚0 disk.md0.out‚ 162.628906252· ;’¤‚(:‡ êüõÒÒ”Ì gpu.0.gpu‚22.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.695' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.565 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Njй_¥‚(:Ô ùüõÒòÙÙË% disk.nvme1n1.out‚19046.4189453125 disk.dm-1.in‚0" disk.dm-1.out‚8195.82861328125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚19046.4189453125 network.sent‚ 9.289779e+06) disk./.usagePercent‚76.00453396492645 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚30100.09814453125$ memory_percent‚95.49379348754883* proc.memory.percent‚18.392324447631836 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0 cpu‚25.63428417448634% disk./.usageGB‚1166.8596076965332 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 162.98046875 disk.dm-0.out‚ 40602.609375( disk.nvme0c0n1.out‚30100.09814453125& proc.memory.availableMB‚ 738.296875" proc.memory.rssMB‚ 3013.3984375 network.recv‚ 1.409769e+06Û"%Èž¦‚(:“ ùüõÒÙ®žÌ gpu.0.gpu‚92.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚127.397' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚42.46566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö=§2§- _wandbruntime‚5302  _runtime‚5302ðõ3ǨÁ % _timestamp‚1.7845121285932293e+09 _runtime‚5302.168817463 _step‚482 & lr-AdamW/pg1‚0.0005026721576021368 & lr-AdamW/pg2‚0.0005026721576021368 trainer/global_step‚5899æEpÌ©Æ & lr-AdamW/pg1‚0.0005026721576021368 & lr-AdamW/pg2‚0.0005026721576021368 trainer/global_step‚5899 % _timestamp‚1.7845121285932293e+09 _runtime‚5302.168817463 _step‚482âtÉLÙ2ª- _wandbruntime‚5302  _runtime‚5302Bä\}í'«çO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47701915 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6842684810240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14107956 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚908890295341568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58337259 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚896774068295152 trainer/global_step‚5899 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6832353038421 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6881605421568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚903606080772299 % _timestamp‚1.7845121285942023e+09 _step‚483 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚908890295341568 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47701917 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106029031 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6842343386112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚196207924718080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚908890653805056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902047951955456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚902047968994816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10293837 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203089548751872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24402092 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24401793 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6881491763712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47701917 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58327116 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6842343386112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106039176 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚5302.170079606 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚896774085334512 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚908890653805056 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚902047951955456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6842684810240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58327116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14108252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47701915 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58337259 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6832012477147 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180®QÝò'¬ìO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚908890653805056 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚902047951955456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6842684810240 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58327116 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14108252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47701915 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58337259 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6832012477147 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47701915 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6842684810240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14107956 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚908890295341568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58337259 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚896774068295152 trainer/global_step‚5899 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6832353038421 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6881605421568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚903606080772299 % _timestamp‚1.7845121285942023e+09 _step‚483 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚908890295341568 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47701917 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106029031 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6842343386112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚196207924718080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚908890653805056 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902047951955456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚902047968994816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10293837 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203089548751872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24402092 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24401793 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6881491763712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47701917 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58327116 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6842343386112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106039176 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚5302.170079606 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚896774085334512ã&Ã2­- _wandbruntime‚5302  _runtime‚5302+0;É{"®ñM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24405847 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106046480 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚903754977428769 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106046480 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6833120865399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚196241123600384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚909040038413312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106056639 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47709710 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _step‚484 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10295574 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚909040038413312 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14110276 trainer/global_step‚5899 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚203123840462336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203123714490880 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚903754619828023 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚896921498962624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5302.791223724 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚902196226457600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58346929 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6843453485056 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6833461426785 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6882609502720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902196226457600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24406143 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14110569 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58336772 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10295571 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6882716861952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106056639 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6843794916352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚902196243496960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6843453485056 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚896921516001984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6843794916352 VHDeviceStatsMonitor.on_train_batch_eg÷ÓÁ|nd/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47709708 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47709708 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58346929 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58336772 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296£k“‹ü&¯öM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106056639 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6843794916352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚902196243496960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6843453485056 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚896921516001984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6843794916352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47709708 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47709708 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58346929 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58336772 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24405847 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106046480 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚903754977428769 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106046480 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6833120865399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚196241123600384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚909040038413312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106056639 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47709710 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _step‚484 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10295574 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚909040038413312 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14110276 trainer/global_step‚5899 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚203123840462336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203123714490880 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚903754619828023 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚896921498962624 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5302.791223724 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚902196226457600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58346929 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6843453485056 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6833461426785 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6882609502720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902196226457600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24406143 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14110569 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58336772 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10295571 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6882716861952äp Õ%2°- _wandbruntime‚5302  _runtime‚5302PLí¥±‚ 2loss/FBt1/assignment_loss‚0.016183655027344302 2loss/FBt2/assignment_loss‚0.003454691751403437 /loss/FRt2/detection_loss‚1.5691926058187675 0loss/FRt1/detection_loss‚0.45436482300815945 /loss/SRqqt2/detection_loss‚0.75541519163845 0loss/FBt1/detection_loss‚0.45311399182619555 trainer/global_step‚5899 0loss/FRt2/assignment_loss‚0.8316085835544407 2loss/SRqqt1/assignment_loss‚0.5661787267675542 0loss/SRqqt1/detection_loss‚0.653244713547243 %loss/total_loss‚8.145041866930477 $ _timestamp‚1.784512129216495e+09 _step‚485 0loss/FRt1/assignment_loss‚1.7337200648431088 /loss/FBt2/detection_loss‚0.5525698272718323 epoch‚14 _runtime‚5302.791786965 2loss/SRqqt2/assignment_loss‚0.5559949918759786¿Ñ㜠²‡ 2loss/SRqqt1/assignment_loss‚0.5661787267675542 0loss/SRqqt1/detection_loss‚0.653244713547243 %loss/total_loss‚8.145041866930477 $ _timestamp‚1.784512129216495e+09 _step‚485 0loss/FRt1/assignment_loss‚1.7337200648431088 /loss/FBt2/detection_loss‚0.5525698272718323 epoch‚14 _runtime‚5302.791786965 2loss/SRqqt2/assignment_loss‚0.5559949918759786 2loss/FBt1/assignment_loss‚0.016183655027344302 2loss/FBt2/assignment_loss‚0.003454691751403437 /loss/FRt2/detection_loss‚1.5691926058187675 0loss/FRt1/detection_loss‚0.45436482300815945 /loss/SRqqt2/detection_loss‚0.75541519163845 0loss/FBt1/detection_loss‚0.45311399182619555 trainer/global_step‚5899 0loss/FRt2/assignment_loss‚0.8316085835544407å…—ó*_³‚(:Ô ˆýõÒŸ¢—Ë network.sent‚ 9.291784e+06& disk.nvme1c1n1.out‚19049.060546875$ disk.nvme1n1.out‚19049.060546875" disk.dm-1.out‚8195.90283203125 proc.cpu.threads‚17) disk./.usagePercent‚75.98907242107035% disk./.usageGB‚1166.6222343444824' disk.nvme0c0n1.out‚30123.7294921875 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚30123.7294921875 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.md0.out‚ 163.390625 disk.dm-1.in‚0& proc.memory.availableMB‚ 738.515625" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836 cpu‚25.633817388331078 network.recv‚ 1.412294e+06 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚40627.93115234375$ memory_percent‚95.49245834350586A‹Û%“´‚(:ˆ ˆýõÒ¯» Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚285.825' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á³Ìcµ‚(:Ø —ýõÒŒöË network.sent‚ 9.356055e+06 disk.nvme0n1.in‚ 0.00390625' proc.memory.availableMB‚ 738.3046875 proc.cpu.threads‚17 network.recv‚ 1.417766e+06 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚19049.63330078125& disk.nvme1n1.out‚19049.63330078125 disk.md0.in‚0 disk.md0.out‚ 163.7421875$ memory_percent‚95.49374580383301% disk./.usageGB‚1166.6222534179688' disk.nvme0c0n1.out‚30124.8310546875% disk.nvme0n1.out‚30124.8310546875 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚40628.72802734375 disk.dm-1.in‚0! disk.dm-1.out‚8196.0146484375( disk./.usagePercent‚75.9890736634404 disk.rbd0.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836 cpu‚25.633340496589717؇Cqœ¶‚(:‘ —ýõÒû°Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.395' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.79833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z]ÏRd·‚(:Ù ¦ýõÒÀмË# disk.dm-0.out‚40639.96240234375$ memory_percent‚96.99671268463135( proc.memory.availableMB‚ 492.05859375( disk.nvme1c1n1.out‚19051.84033203125 disk.md0.in‚0' disk.nvme0c0n1.out‚30134.9521484375" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836 disk.dm-1.in‚0! disk.dm-1.out‚8196.0771484375$ disk./.usageGB‚1166.622200012207 disk.md0.out‚ 164.22265625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚30134.9521484375 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚19051.84033203125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 cpu‚25.63278490615666 network.sent‚ 9.359078e+06 network.recv‚ 1.421228e+06) disk./.usagePercent‚75.98907018480423 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625å¸ ž¸‚(:“ ¦ýõÒ΢Ì gpu.0.gpu‚51.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚259.898' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.63266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òß2¹- _wandbruntime‚5345  _runtime‚5345· àƺÀ trainer/global_step‚5949 $ _timestamp‚1.784512172153272e+09 _runtime‚5345.729674447 _step‚486 & lr-AdamW/pg1‚0.0005007658711218798 & lr-AdamW/pg2‚0.0005007658711218798§•E˻Š$ _timestamp‚1.784512172153272e+09 _runtime‚5345.729674447 _step‚486 & lr-AdamW/pg1‚0.0005007658711218798 & lr-AdamW/pg2‚0.0005007658711218798 trainer/global_step‚5949æn tÝ2¼- _wandbruntime‚5345  _runtime‚5345Lá ì'½æO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58811768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58821911 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10380540 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 $ _timestamp‚1.784512172153954e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24604309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14223769 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14223483 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197866956834304 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚911033454959941 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚916359589550592 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106903333 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916359589550592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204804274477056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚909461677062656 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6887495690555 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24604020 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6937336254976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6897912487936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6898253912064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6887836251829 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6937441524224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚909461694102016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚911033097359307 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48091567 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚904145601668752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6897912487936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚480915656€UÍö¾ëO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6937441524224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚909461694102016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚911033097359307 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48091567 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚904145601668752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6897912487936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48091565 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58811768 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58821911 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10380540 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 $ _timestamp‚1.784512172153954e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24604309 XKDeviceStat}¸ÒKûsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14223769 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14223483 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197866956834304 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚911033454959941 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚916359589550592 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106903333 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916359589550592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204804274477056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚909461677062656 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6887495690555 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24604020 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6937336254976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6897912487936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6898253912064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6887836251829çjbM¡2¿- _wandbruntime‚5346  _runtime‚5346¡Ä¶•ö&ÀðM FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106930941 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204838576245248 % _timestamp‚1.7845121727746608e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚904293032336224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58831581 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10382271 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _runtime‚ 5346.35036211 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚916509332622336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48099358 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚916508974151680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48099360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106920782 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48099360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58821424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚916509332622336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10382274 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197900118492160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚911181636415031 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚909609968604160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24608356 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 trainer/global_step‚5949 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197900137104384 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58821424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106920782 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48099358 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106930941 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6899022586880 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24608067 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204838695925248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚904293049375584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚909609951564800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6938457753088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6899364018176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6899022586880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2kâwVû&ÁõM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106930941 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6899022586880 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24608067 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204838695925248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚904293049375584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚909609951564800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6938457753088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6899364018176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6899022586880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106930941 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204838576245248 % _timestamp‚1.7845121727746608e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚904293032336224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58831581 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10382271 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _runtime‚ 5346.35036211 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚916509332622336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48099358 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚916508974151680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48099360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106920782 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48099360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58821424 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚916509332622336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10382274 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197900118492160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚911181636415031 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚909609968604160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24608356 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 trainer/global_step‚5949 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197900137104384 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58821424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106920782 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48099358 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746èÜ¢2Â- _wandbruntime‚5346  _runtime‚5346âÐ][ à 2loss/SRqqt1/assignment_loss‚0.4944170761926106 1loss/SRqqt2/assignment_loss‚0.554705151133755 trainer/global_step‚5949 0loss/FRt2/assignment_loss‚0.7081369038234625 1loss/SRqqt2/detection_loss‚0.6688729336030745 $ _timestamp‚1.784512172775712e+09 0loss/FRt1/assignment_loss‚1.6881460999937186 3loss/FBt2/assignment_loss‚0.0013011885362893257 /loss/FRt2/detection_loss‚1.5254822383086637 1loss/SRqqt1/detection_loss‚0.6810307144622478 _runtime‚5346.351014761 2loss/FBt1/assignment_loss‚0.009703810021392926 /loss/FRt1/detection_loss‚0.5083595407960942 /loss/FBt1/detection_loss‚0.4946421082179081 /loss/FBt2/detection_loss‚0.7043053274729142 %loss/total_loss‚8.039103092562131 epoch‚14 _step‚489L+ËSĈ 2loss/SRqqt1/assignment_loss‚0.4944170761926106 1loss/SRqqt2/assignment_loss‚0.554705151133755 trainer/global_step‚5949 0loss/FRt2/assignment_loss‚0.7081369038234625 1loss/SRqqt2/detection_loss‚0.6688729336030745 $ _timestamp‚1.784512172775712e+09 0loss/FRt1/assignment_loss‚1.6881460999937186 3loss/FBt2/assignment_loss‚0.0013011885362893257 /loss/FRt2/detection_loss‚1.5254822383086637 1loss/SRqqt1/detection_loss‚0.6810307144622478 _runtime‚5346.351014761 2loss/FBt1/assignment_loss‚0.009703810021392926 /loss/FRt1/detection_loss‚0.5083595407960942 /loss/FBt1/detection_loss‚0.4946421082179081 /loss/FBt2/detection_loss‚0.7043053274729142 %loss/total_loss‚8.039103092562131 epoch‚14 _step‚489éÇ9ÜcÅ‚(:Ø µýõÒˆ˜ëË network.sent‚ 9.377857e+06 network.recv‚ 1.425001e+06 disk.rbd0.in‚0 disk.md0.out‚ 164.609375 cpu‚25.63231741383646 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚30136.0400390625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚19052.46826171875$ memory_percent‚96.99690341949463( proc.memory.availableMB‚ 492.02734375 proc.cpu.threads‚17% disk./.usageGB‚1166.6221923828125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚19052.46826171875 disk.nvme1n1.in‚0" proc.memory.rssMB‚ 3013.3984375) disk./.usagePercent‚75.98906968785622 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚40640.70361328125! disk.dm-1.out‚8196.2177734375' disk.nvme0c0n1.out‚30136.0400390625 disk.md0.in‚0* proc.memory.percent‚18.392324447631836µŽ ìŸÆ‚(:” µýõÒÈõ¨Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.798' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.266000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@UüßbÇ‚(:× ÄýõÒƒ¾¼Ë( proc.memory.availableMB‚ 492.01953125 network.sent‚ 9.428221e+06% disk.nvme0c0n1.out‚30138.35546875 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚19054.99560546875 disk.nvme1n1.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚8197.7255859375* proc.memory.percent‚18.392324447631836 network.recv‚ 1.430216e+06) disk./.usagePercent‚75.98913578194335# disk.nvme0n1.out‚30138.35546875 disk.nvme1c1n1.in‚0" proc.memory.rssMB‚ 3013.3984375 proc.cpu.threads‚17& disk.nvme1n1.out‚19054.99560546875 disk.md0.out‚ 164.87109375 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚40643.45263671875$ memory_percent‚96.99695110321045 cpu‚25.631813235433718% disk./.usageGB‚1166.6232070922852 disk.nvme0c0n1.in‚ 0.00390625 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0+’Ï‘È‚(:† ÄýõÒÄóôË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.505' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.835 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê”íbÉ‚(:× ÓýõÒÝæ®Ë% disk./.usageGB‚1166.8605346679688' disk.nvme0c0n1.out‚30240.6572265625 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚19090.9365234375 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚40778.84326171875* proc.memory.percent‚18.392324447631836 network.sent‚ 9.43312e+06 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.md0.in‚0' proc.memory.availableMB‚ 492.0234375" proc.memory.rssMB‚ 3013.3984375 network.recv‚ 1.433685e+06( disk./.usagePercent‚76.0045943441113" disk.dm-1.out‚8199.56787109375 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme1n1.out‚19090.9365234375 disk.md0.out‚ 165.33203125$ memory_percent‚96.99692726135254 cpu‚25.631401947988063% disk.nvme0n1.out‚30240.6572265625 disk.nvme1n1.in‚0 proc.cpu.threads‚17¾×±ížÊ‚(:“ ÓýõÒßæìË gpu.0.gpu‚73.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚270.194' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.06466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eâýû2Ë- _wandbruntime‚5389  _runtime‚5389‡üÑ¢ÅÌ¿ % lr-AdamW/pg2‚0.000498847155139128 trainer/global_step‚5999 % _timestamp‚1.7845122156896837e+09 _runtime‚5389.265279272 _step‚490 % lr-AdamW/pg1‚0.000498847155139128Z!ÔÆÊÍÄ _runtime‚5389.265279272 _step‚490 % lr-AdamW/pg1‚0.000498847155139128 % lr-AdamW/pg2‚0.000498847155139128 trainer/global_step‚5999 % _timestamp‚1.7845122156896837e+09êù{V$2Î- _wandbruntime‚5389  _runtime‚5389Ò­k3ÏæO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206519266421760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10467237 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚923828883759616 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6942978903963 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10467240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107787780 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6993296083456 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚911517152081712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚923829242223104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 MFDeviceStatsMonitor.on_train_batch_st7l/¹ art/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6953481589760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚916875419209216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24806618 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48481215 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚916875419209216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6953481589760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59296420 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107777635 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚923828883759616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚911517135042352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚5999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59306563 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 % _timestamp‚1.7845122156907363e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48481217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _runtime‚5389.266352444 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚199525970338304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48481215 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59306563 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚916875402169856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6953823013888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24806893 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6993201299968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48481215 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚916875419209216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6953481589760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59296420 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107777635 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚923828883759616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚911517135042352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚5999 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59306563 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 % _timestamp‚1.7845122156907363e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48481217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _runtime‚5389.266352444 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚199525970338304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48481215 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59306563 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚916875402169856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6953823013888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24806893 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6993201299968 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10467240ëa d2Ñ- _wandbruntime‚5389  _runtime‚5389x Wõ&ÒïM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24810678 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10468974 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚918608653002039 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107805243 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59316233 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107795084 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚911664582749184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚917023676672000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107805243 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206553561451520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59306076 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206553445965824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59316233 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10468971 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚923978626831360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14341707 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚917023676672000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48489008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6994410843136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚917023693711360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48489010 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6944427853601 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199559150608384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6954591688704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24810944 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚917023693711360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ý+*ƒ ÓôM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6944427853601 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199559150608384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6954591688704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24810944 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚917023693711360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24810678 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10468974 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚918608653002039 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107805243 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59316233 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107795084 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peÿ%·îak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚911664582749184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚917023676672000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107805243 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206553561451520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59306076 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206553445965824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59316233 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10468971 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚923978626831360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14341707 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚917023676672000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48489008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6994410843136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚917023693711360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48489010ìý’ÏØ2Ô- _wandbruntime‚5389  _runtime‚5389¯“l Õ… 0loss/FRt2/assignment_loss‚0.7127382558526345 3loss/FBt2/assignment_loss‚0.0008933406219196851 /loss/FRt1/detection_loss‚0.5102679725356123 trainer/global_step‚5999 1loss/SRqqt1/detection_loss‚0.7499339256857126 2loss/SRqqt1/assignment_loss‚0.4386731314086065 2loss/SRqqt2/assignment_loss‚0.4948699814785422 3loss/FBt1/assignment_loss‚0.0006471596853884688 /loss/FRt2/detection_loss‚1.5242458018067393 /loss/FBt2/detection_loss‚0.4835231025017014 %loss/total_loss‚7.473502455344596 $ _timestamp‚1.784512216315109e+09 _step‚493 0loss/FRt1/assignment_loss‚1.5565873297094142 0loss/SRqqt2/detection_loss‚0.601787516656419 0loss/FBt1/detection_loss‚0.39933493740190534 epoch‚14 _runtime‚5389.890434869€ó¢áÖŠ 2loss/SRqqt1/assignment_loss‚0.4386731314086065 2loss/SRqqt2/assignment_loss‚0.4948699814785422 3loss/FBt1/assignment_loss‚0.0006471596853884688 /loss/FRt2/detection_loss‚1.5242458018067393 /loss/FBt2/detection_loss‚0.4835231025017014 %loss/total_loss‚7.473502455344596 $ _timestamp‚1.784512216315109e+09 _step‚493 0loss/FRt1/assignment_loss‚1.5565873297094142 0loss/SRqqt2/detection_loss‚0.601787516656419 0loss/FBt1/detection_loss‚0.39933493740190534 epoch‚14 _runtime‚5389.890434869 0loss/FRt2/assignment_loss‚0.7127382558526345 3loss/FBt2/assignment_loss‚0.0008933406219196851 /loss/FRt1/detection_loss‚0.5102679725356123 trainer/global_step‚5999 1loss/SRqqt1/detection_loss‚0.7499339256857126íG`Ãcׂ(:Ø âýõÒ’·ÓË) disk./.usagePercent‚76.00459285326723! disk.dm-1.out‚8199.7763671875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚19121.5791015625$ memory_percent‚96.99733257293701" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836 network.sent‚ 9.448927e+06 network.recv‚ 1.436962e+06" disk.dm-0.out‚40891.9951171875 disk.dm-1.in‚0% disk.nvme0n1.out‚30324.8603515625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚19121.5791015625 disk.md0.out‚ 166.04296875% disk./.usageGB‚1166.8605117797852 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚30324.8603515625( proc.memory.availableMB‚ 491.95703125 cpu‚25.635743273872787 proc.cpu.threads‚17 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.md0.in‚0E£î‘Ø‚(:† âýõÒŽä’Ì gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚53.0 gpu.0.powerWatts‚118.563' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.521 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t Ç2Ù- _wandbruntime‚5406  _runtime‚5406äâ‘‹®=Ú¨{ ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.018764501437544823 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 /particle/accuracy_4_of_4‚0.9621089100837708 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6719205379486084 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7931390404701233 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6594711542129517 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5570833086967468 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9897685050964355 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.996985137462616 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_1_of_4‚0.9625982642173767 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12389250844717026 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_2_of_3‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976779222488403 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7518237233161926 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22425000369548798 +particle/specificity‚0.9060608744621277 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1923026740550995 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2767130136489868 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19765645265579224 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 )loss/val_total_loss‚7.882274714766442 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8524227142333984 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3229166567325592 particle/accuracy_3_of_5‚1 % _timestamp‚1.7845122324742224e+09 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7494680285453796 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19910864531993866 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.038142137229442596 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3889603912830353 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6594711542129517 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6563354730606079 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010455606505274773 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6588330268859863 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9901471138000488 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7930342555046082 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.046296652406454086 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_2_of_4‚0.7778429985046387 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7055927515029907 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5856872797012329 jet/accuracy_4_of_5‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7505143284797668 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1555379033088684 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6656358242034912 +jet/accuracy_3_of_4‚0.09568073600530624 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9020311236381531 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5092087388038635 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.45522552728652954 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7433097958564758 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8953602313995361 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9581507444381714 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7253255248069763 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6392761468887329 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.15027284622192383 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8921011686325073 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1415630728006363 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24487851560115814 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17801974713802338 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 _step‚494 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02917565405368805 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988561272621155 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9969152808189392 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19780376553535461 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_4_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31454575061798096 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963561296463013 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 *jet/accuracy_1_of_3‚0.9285111427307129 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6581836342811584 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_1‚0.3276902437210083 *jet/accuracy_2_of_3‚0.5509746074676514 particle/accuracy_1_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5786693096160889 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.044082723557949066 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 jet/accuracy_2_of_5‚1 jet/accuracy_5_of_5‚0 trainer/global_step‚5999 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2039066106081009 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6600901484489441 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7000817656517029 +jet/accuracy_3_of_3‚0.07999034970998764 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4685322940349579 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.9051020741462708 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7686408758163452 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8264172077178955 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9110544323921204 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6625670790672302 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20730173587799072 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8967120051383972 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1317499428987503 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8234736919403076 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5541809797286987 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2703710198402405 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941260814666748 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20569324493408203 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9111487865447998 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1778368353843689 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04240167513489723 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38630375266075134 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7698897123336792 particle/accuracy_2_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2186916470527649 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17949244379997253 particle/accuracy_3_of_3‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6788409948348999 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49156615138053894 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12916667759418488 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.3720627427101135 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7198963761329651 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6819992065429688 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7658455967903137 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7034440040588379 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04379383847117424 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.33706995844841003 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6523194313049316 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6458333134651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6737082600593567 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6333276629447937 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988489747047424 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9974282383918762 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7521494626998901 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8234736919403076 *jet/accuracy_1_of_2‚0.8017646074295044 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7506623268127441 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8634017705917358 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 epoch‚14 _runtime‚5406.050294553 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965829253196716 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019677886739373207 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22162137925624847 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45370373129844666 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7517220973968506 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11478898674249649 jet/accuracy_1_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21940529346466064 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20569324493408203 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19160309433937073 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6990649104118347 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7450917959213257 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45480942726135254 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17836067080497742 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.038142137229442596 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.701233446598053 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20778383314609528 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.662529706954956 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 +particle/sensitivity‚0.6562795042991638 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.633104145526886 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4875057637691498 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03563055396080017 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16765519976615906 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04582883417606354uF4¶²Û­{ :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45480942726135254 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17836067080497742 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.038142137229442596 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.701233446598053 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20778383314609528 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.662529706954956 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 +particle/sensitivity‚0.6562795042991638 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.633104145526886 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4875057637691498 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03563055396080017 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16765519976615906 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04582883417606354 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.018764501437544823 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 /particle/accuracy_4_of_4‚0.9621089100837708 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6719205379486084 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7931390404701233 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6594711542129517 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5570833086967468 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9897685050964355 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.996985137462616 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_1_of_4‚0.9625982642173767 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12389250844717026 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_2_of_3‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976779222488403 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7518237233161926 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22425000369548798 +particle/specificity‚0.9060608744621277 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1923026740550995 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2767130136489868 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19765645265579224 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 )loss/val_total_loss‚7.882274714766442 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8524227142333984 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3229166567325592 particle/accuracy_3_of_5‚1 % _timestamp‚1.7845122324742224e+09 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7494680285453796 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19910864531993866 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.038142137229442596 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3889603912830353 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6594711542129517 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6563354730606079 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010455606505274773 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6588330268859863 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9901471138000488 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7930342555046082 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.046296652406454086 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_2_of_4‚0.7778429985046387 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7055927515029907 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5856872797012329 jet/accuracy_4_of_5‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7505143284797668 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1555379033088684 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6656358242034912 +jet/accuracy_3_of_4‚0.09568073600530624 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9020311236381531 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5092087388038635 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.45522552728652954 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7433097958564758 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8953602313995361 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9581507444381714 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7253255248069763 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6392761468887329 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.15027284622192383 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8921011686325073 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1415630728006363 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24487851560115814 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17801974713802338 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 _step‚494 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02917565405368805 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988561272621155 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9969152808189392 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19780376553535461 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/Ÿ÷4FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_4_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31454575061798096 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963561296463013 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 *jet/accuracy_1_of_3‚0.9285111427307129 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6581836342811584 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_1‚0.3276902437210083 *jet/accuracy_2_of_3‚0.5509746074676514 particle/accuracy_1_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5786693096160889 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.044082723557949066 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 jet/accuracy_2_of_5‚1 jet/accuracy_5_of_5‚0 trainer/global_step‚5999 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2039066106081009 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6600901484489441 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7000817656517029 +jet/accuracy_3_of_3‚0.07999034970998764 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4685322940349579 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.9051020741462708 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7686408758163452 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8264172077178955 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9110544323921204 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6625670790672302 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20730173587799072 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8967120051383972 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1317499428987503 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8234736919403076 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5541809797286987 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2703710198402405 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941260814666748 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20569324493408203 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9111487865447998 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1778368353843689 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04240167513489723 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38630375266075134 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7698897123336792 particle/accuracy_2_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2186916470527649 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17949244379997253 particle/accuracy_3_of_3‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6788409948348999 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49156615138053894 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12916667759418488 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 6validation_average_jet_accuracy‚0.3720627427101135 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7198963761329651 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6819992065429688 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7658455967903137 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7034440040588379 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04379383847117424 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.33706995844841003 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6523194313049316 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6458333134651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6737082600593567 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6333276629447937 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988489747047424 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9974282383918762 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7521494626998901 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8234736919403076 *jet/accuracy_1_of_2‚0.8017646074295044 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7506623268127441 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8634017705917358 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 epoch‚14 _runtime‚5406.050294553 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965829253196716 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019677886739373207 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22162137925624847 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45370373129844666 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7517220973968506 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11478898674249649 jet/accuracy_1_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21940529346466064 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20569324493408203 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19160309433937073 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6990649104118347 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7450917959213257îÝ`ø8dÜ‚(:Ù ñýõÒª°®Ë disk.dm-1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚40892.5576171875! disk.dm-1.out‚8445.2177734375$ memory_percent‚95.49150466918945& proc.memory.availableMB‚ 738.671875* proc.memory.percent‚18.392324447631836( disk.nvme0c0n1.out‚30325.66162109375 cpu‚25.63771879763613 network.sent‚ 9.498481e+06 network.recv‚ 1.442184e+06 disk.md0.out‚ 166.34765625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625" proc.memory.rssMB‚ 3013.3984375 proc.cpu.threads‚17) disk./.usagePercent‚76.00453222560836% disk./.usageGB‚1166.8595809936523 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚30325.66162109375 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚19367.43896484375& disk.nvme1n1.out‚19367.43896484375*§§žÝ‚(:“ ñýõÒ¢Æ˜Ì gpu.0.gpu‚97.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚265.543' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.51433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï ©daÞ‚(:Ö €þõÒ¼ÕËË disk.md0.out‚ 166.578125 disk.dm-1.in‚0& proc.memory.availableMB‚ 738.671875 proc.cpu.threads‚17 network.sent‚ 9.557327e+06 network.recv‚ 1.447245e+06 disk.rbd0.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚8501.46240234375" proc.memory.rssMB‚ 3013.3984375 cpu‚25.637199513658114$ disk./.usageGB‚1166.859619140625 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚19425.7783203125" disk.dm-0.out‚40905.0537109375$ memory_percent‚95.49150466918945* proc.memory.percent‚18.392324447631836( disk.nvme0c0n1.out‚30336.57275390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0% disk.nvme1n1.out‚19425.7783203125) disk./.usagePercent‚76.00453471034848 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚30336.572753906254p²§Ÿß‚(:” €þõÒÅ´†Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚121.871' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.623666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€@¦[^à‚(:Ó þõÒÜÙ¢Ë) disk./.usagePercent‚76.00453471034848 disk.dm-1.in‚0" disk.dm-1.out‚8690.86083984375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625& disk.nvme1c1n1.out‚19615.626953125 disk.dm-0.in‚ 0.00390625* proc.memory.percent‚18.392324447631836 network.recv‚ 1.45101e+06 disk.rbd0.in‚0( disk.nvme0c0n1.out‚30337.53857421875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 738.61328125" proc.memory.rssMB‚ 3013.3984375 network.sent‚ 9.560249e+06& disk.nvme0n1.out‚30337.53857421875 disk.md0.in‚0 cpu‚25.636780066911587 proc.cpu.threads‚17$ disk./.usageGB‚1166.859619140625 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme1n1.out‚19615.626953125 disk.md0.out‚ 166.90625" disk.dm-0.out‚40905.7412109375# memory_percent‚95.4918622970581"K~ùŸá‚(:” þõÒÆ—ŒÍ gpu.0.gpu‚80.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚119.648' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.882666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×tdÄ2â- _wandbruntime‚5449  _runtime‚5449Š©ÇnÇãÁ & lr-AdamW/pg1‚0.0004969161328894236 & lr-AdamW/pg2‚0.0004969161328894236 trainer/global_step‚6049 % _timestamp‚1.7845122763813665e+09 _runtime‚5449.958019377 _step‚495¶Tu ÌäÆ & lr-AdamW/pg1‚0.0004969161328894236 & lr-AdamW/pg2‚0.0004969161328894236 trainer/global_step‚6049 % _timestamp‚1.7845122763813665e+09 _runtime‚5449.958019377 _step‚495ï,µÞ2å- _wandbruntime‚5449  _runtime‚5449•éuë'æåO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59872714 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚933261198146048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208277415733760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚927829047991083 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7018765966273 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7029368529920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚926231488192000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚933261198146048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14487643 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚6049 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7029368529920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚5449.958124147 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚926231488192000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚927828690390449 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚926231471152640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48993673 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14487912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7029709954048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10558056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚933260839682560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚920809924424176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7019106527547 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25045968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10558053 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360ßê2ð'çêO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚920809924424176 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7019106527547 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25045968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10558053 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59872714 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚933261198146048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208277415733760 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚927829047991083 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7018765966273 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7029368529920 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚926231488192000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚933261198146048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14487643 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚6049 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7029368529920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚5449.958124147 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚926231488192000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚927828690390449 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚926231471152640 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48993673 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14487912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7029709954048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10558056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚933260839682560 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452ð!¦ã§2è- _wandbruntime‚5450  _runtime‚5450ÆÙ“éðM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201240742774784 =7DeviceStatsMonitor.on_train_batch_eníþ­c&d/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚933410582754304 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚926379762694144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10559787 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208311601638912 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59872227 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49001466 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108873693 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7030478628864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108883852 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108873693 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14489955 % _timestamp‚1.7845122773205123e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108883852 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25049742 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚6049 _step‚497 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7030820060160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚926379745654784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚927977229446173 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201240724162560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7030820060160 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7019874354525 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108883852 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25049742 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚6049 _step‚497 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7030820060160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚926379745654784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚927977229446173 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201240724162560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7030820060160 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7019874354525 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚933410582754304 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚926379762694144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10559787 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208311601638912 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59872227 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49001466 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108873693 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7030478628864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108883852 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108873693 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14489955 % _timestamp‚1.7845122773205123e+09ñUÑzí2ë- _wandbruntime‚5450  _runtime‚5450Ÿ?4 ìƒ /loss/FRt2/assignment_loss‚0.824627660636632 %loss/total_loss‚8.271039999874688 _runtime‚5450.897031285 2loss/FBt1/assignment_loss‚0.009620855969925547 2loss/FBt2/assignment_loss‚0.001037422481908079 0loss/FRt1/detection_loss‚0.49308358824734083 1loss/SRqqt1/detection_loss‚0.6817231209617367 /loss/FBt2/detection_loss‚0.8553439937531948 trainer/global_step‚6049 0loss/FRt1/assignment_loss‚1.6543021872288546 2loss/SRqqt1/assignment_loss‚0.5623049065750875 2loss/SRqqt2/assignment_loss‚0.6094110595794082 0loss/SRqqt2/detection_loss‚0.631407127626932 /loss/FBt1/detection_loss‚0.3549909922955054 epoch‚15 /loss/FRt2/detection_loss‚1.5931870845181622 % _timestamp‚1.7845122773217368e+09 _step‚498É?–íˆ /loss/FRt2/detection_loss‚1.5931870845181622 % _timestamp‚1.7845122773217368e+09 _step‚498 /loss/FRt2/assignment_loss‚0.824627660636632 %loss/total_loss‚8.271039999874688 _runtime‚5450.897031285 2loss/FBt1/assignment_loss‚0.009620855969925547 2loss/FBt2/assignment_loss‚0.001037422481908079 0loss/FRt1/detection_loss‚0.49308358824734083 1loss/SRqqt1/detection_loss‚0.6817231209617367 /loss/FBt2/detection_loss‚0.8553439937531948 trainer/global_step‚6049 0loss/FRt1/assignment_loss‚1.6543021872288546 2loss/SRqqt1/assignment_loss‚0.5623049065750875 2loss/SRqqt2/assignment_loss‚0.6094110595794082 0loss/SRqqt2/detection_loss‚0.631407127626932 /loss/FBt1/detection_loss‚0.3549909922955054 epoch‚15ò)2‡eRî‚(:Ç žþõÒËÔË* proc.memory.percent‚18.392324447631836 cpu‚25.636218965929 proc.cpu.threads‚17 disk.dm-0.out‚ 40907.3046875 network.sent‚ 9.579021e+06 network.recv‚ 1.454202e+06 disk.nvme0n1.in‚ 0.00390625 disk.md0.in‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 30339.2734375$ memory_percent‚95.49355506896973$ disk./.usageGB‚1166.859676361084 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.md0.out‚ 167.33984375) disk./.usagePercent‚76.00453843745866" disk.nvme0n1.out‚ 30339.2734375& disk.nvme1c1n1.out‚19616.595703125$ disk.nvme1n1.out‚19616.595703125 disk.dm-1.out‚8691.056640625 disk.rbd0.out‚0" proc.memory.rssMB‚ 3013.3984375 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625' proc.memory.availableMB‚ 738.3359375Ȳ ÷Ÿï‚(:” žþõÒƒå˜Ì gpu.0.gpu‚83.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚179.162' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚59.720666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¼ Õ^ð‚(:Ó ­þõÒ»¬æË disk.md0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚8691.095703125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625 network.recv‚ 1.458803e+06) disk./.usagePercent‚76.00452800155017 disk.nvme1n1.in‚0 disk.md0.out‚ 167.78125 disk.dm-0.out‚40907.89453125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚30340.24072265625 network.sent‚ 9.628471e+06 disk.dm-0.in‚ 0.00390625( disk.nvme1c1n1.out‚19617.19970703125# memory_percent‚95.4920768737793* proc.memory.percent‚18.392324447631836 cpu‚25.635708916485235& disk.nvme0n1.out‚30340.24072265625 disk.nvme1c1n1.in‚0& proc.memory.availableMB‚ 738.578125" proc.memory.rssMB‚ 3013.3984375 proc.cpu.threads‚17% disk./.usageGB‚1166.8595161437988& disk.nvme1n1.out‚19617.19970703125E—ËAžñ‚(:“ ­þõÒßí¦Ì gpu.0.gpu‚95.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚264.868' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.28933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gcÁwaò‚(:Ö ¼þõÒ…µÈË network.sent‚ 9.631595e+06 network.recv‚ 1.462145e+06) disk./.usagePercent‚76.00454117067278$ disk./.usageGB‚1166.859718322754& disk.nvme1c1n1.out‚19617.783203125 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0" disk.dm-1.out‚8691.19384765625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚30342.44189453125( proc.memory.availableMB‚ 738.54296875* proc.memory.percent‚18.392324447631836 cpu‚25.635250719732593 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚19617.783203125" disk.dm-0.out‚40910.0009765625 proc.cpu.threads‚17 disk.md0.out‚ 168.046875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚30342.44189453125$ memory_percent‚95.49229145050049" proc.memory.rssMB‚ 3013.3984375@¨q¶ó‚(:’ ¼þõÒˆ“›Ì gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚268.66' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.55333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|ÿVc2ô- _wandbruntime‚5493  _runtime‚5493\iÖØÅõ¿ % lr-AdamW/pg1‚0.000494972928398717 % lr-AdamW/pg2‚0.000494972928398717 trainer/global_step‚6099 % _timestamp‚1.7845123200046146e+09 _runtime‚5493.580212923 _step‚499ͧ/‹ÊöÄ _step‚499 % lr-AdamW/pg1‚0.000494972928398717 % lr-AdamW/pg2‚0.000494972928398717 trainer/global_step‚6099 % _timestamp‚1.7845123200046146e+09 _runtime‚5493.580212923óˆÏ)2÷- _wandbruntime‚5493  _runtime‚5493ìï)S øæO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202866557396480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚940730492355072 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚940730133891584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚933645196259840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109740691 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49383323 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚928181474837136 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚940730492355072 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚940730133891584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109730546 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚5493.583066218 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7084937631744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚933645196259840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49383325 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7085279055872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60357366 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7084937631744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209992185521152 @:DeviceStatsMonitor.on_train_’pmÏá batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60347223 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49383325 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚933645213299200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25247870 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209992303111168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10644753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10644756 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚500 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚935255706977457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7074589740955 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚940730133891584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚933645196259840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109740691 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49383323 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚928181474837136 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚940730492355072 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚940730133891584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109730546 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚5493.583066218 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7084937631744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚933645196259840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49383325 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7085279055872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60357366 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7084937631744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209992185521152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60347223 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49383325 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚933645213299200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25247870 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209992303111168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10644753 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10644756 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚500 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚935255706977457 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7074589740955 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚501 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14605432 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202899737666560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10646490 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14605709 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7075698129319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7086047730688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10646487 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚210026599949824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7086389161984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7086047730688 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25251919 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚928328905504608 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚933793470761984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚210026480269824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7126742603264 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚935404246033181 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60367036 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49391116 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25252199 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚933793487801344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚940879876963328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49391118 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109747995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 $ _timestamp‚1.784512320627652e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚933793487801344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60356879 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7086389161984 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7126843671040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚6099 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109758154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7075357567933 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224—º°ôã#üõM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109758154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7075357567933 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202899756278784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49391116 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚933793470761984 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚501 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14605432 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202899737666560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10646490 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14605709 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7075698129319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7086047730688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10646487 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚210026599949824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7086389161984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7086047730688 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25251919 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚928328905504608 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚933793470761984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚210026480269824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7126742603264 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚935404246033181 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60367036 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49391116 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25252199 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚933793487801344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚940879876963328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49391118 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109747995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 $ _timestamp‚1.784512320627652e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚933793487801344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60356879 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7086389161984 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7126843671040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚6099 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211707034306048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚935552991171376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60831875 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚948199786564096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚204525589512704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚935553008210736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49772973 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49772973 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚942682723564465 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49772975 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60842018 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚948199786564096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14717788 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚948199428100608 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7129732393089 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7130072954363 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10731456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7140506733568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211707156090368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚941058938406400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49772975 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚942683081165099 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚941058921367040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511¶ßUò'‹ìO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49772975 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚942683081165099 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚941058921367040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211707034306048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚935552991171376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60831875 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚948199786564096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚204525589512704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚935553008210736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49772973 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49772973 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚942682723564465 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49772975 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60842018 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚948199786564096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14717788 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚948199428100608 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7129732393089 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7130072954363 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10731456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7140506733568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211707156090368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚941058938406400 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534øÁ 2Œ- _wandbruntime‚5537  _runtime‚5537m´µåñM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25452941 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚505 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚948348812701696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚942831620220935 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7141616832512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 % _timestamp‚1.7845123643365595e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7141958263808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚204558751170560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚204558769782784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚935700438878208 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60841531 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49780768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10733190 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14719754 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚5537.912331681 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110632456 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7182582324224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6149 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚941207195869184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60851688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚948349171172352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25453234 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7182683392000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110632456D1ü&ŽöM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110632456 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7182582324224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6149 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚941207195869184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60851688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚948349171172352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25453234 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7182683392000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110632456 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25452941 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚505 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚948348812701696 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚942831620220935 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7141616832512 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 % _timestamp‚1.7845123643365595e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7141958263808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚204558751170560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚204558769782784 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚935700438878208 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60841531 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49780768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10733190 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14719754 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚5537.912331681 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ùµ”×2- _wandbruntime‚5537  _runtime‚5537Ë¡H¶ † 3loss/SRqqt1/assignment_loss‚0.40851763314435025 3loss/FBt1/assignment_loss‚0.0006368008549312494 %loss/total_loss‚7.946737049151505 % _timestamp‚1.7845123643385618e+09 _step‚506 /loss/FRt1/assignment_loss‚1.604355337745265 0loss/FRt2/assignment_loss‚0.8271011894898034 1loss/SRqqt2/detection_loss‚0.6710579342818339 .loss/FBt1/detection_loss‚0.463578148876523 2loss/SRqqt2/assignment_loss‚0.5667004567975531 3loss/FBt2/assignment_loss‚0.0005252149778654156 0loss/FRt1/detection_loss‚0.48466240878728817 /loss/FRt2/detection_loss‚1.6392288403461528 1loss/SRqqt1/detection_loss‚0.6974563069312357 /loss/FBt2/detection_loss‚0.5829167769187027 trainer/global_step‚6149 epoch‚15 _runtime‚5537.914002714Ýôøב‹ 3loss/FBt1/assignment_loss‚0.0006368008549312494 %loss/total_loss‚7.946737049151505 % _timestamp‚1.7845123643385618e+09 _step‚506 /loss/FRt1/assignment_loss‚1.604355337745265 0loss/FRt2/assignment_loss‚0.8271011894898034 1loss/SRqqt2/detection_loss‚0.6710579342818339 .loss/FBt1/detection_loss‚0.463578148876523 2loss/SRqqt2/assignment_loss‚0.5667004567975531 3loss/FBt2/assignment_loss‚0.0005252149778654156 0loss/FRt1/detection_loss‚0.48466240878728817 /loss/FRt2/detection_loss‚1.6392288403461528 1loss/SRqqt1/detection_loss‚0.6974563069312357 /loss/FBt2/detection_loss‚0.5829167769187027 trainer/global_step‚6149 epoch‚15 _runtime‚5537.914002714 3loss/SRqqt1/assignment_loss‚0.40851763314435025úœžM¶f’‚(:Û øþõÒ©´™Ë disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚30348.68115234375 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚19620.38232421875& disk.nvme1n1.out‚19620.38232421875 disk.md0.out‚ 169.50390625" disk.dm-0.out‚40915.3212890625 disk.dm-1.in‚0 disk.nvme1n1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚30348.68115234375$ memory_percent‚95.49298286437988" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836 proc.cpu.threads‚17% disk./.usageGB‚1166.8604850769043 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625! disk.dm-1.out‚8691.5537109375 disk.rbd0.in‚0' proc.memory.availableMB‚ 738.4296875 cpu‚25.633347197323097 network.sent‚ 9.709607e+06 network.recv‚ 1.477845e+06) disk./.usagePercent‚76.004591113949157õ$.ž“‚(:“ øþõÒÖ܉Ì gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚279.089' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.02966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.$ÖXf”‚(:Û ‡ÿõÒăË& disk.nvme1n1.out‚19621.07177734375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚30349.73388671875# memory_percent‚95.4901933670044" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836 network.sent‚ 9.772038e+06& disk.nvme0n1.out‚30349.73388671875 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚40915.8876953125! disk.dm-1.out‚8691.6083984375 disk.rbd0.out‚0 cpu‚25.633839621777753 network.recv‚ 1.483092e+06( disk.nvme1c1n1.out‚19621.07177734375 disk.md0.out‚ 170.02734375( proc.memory.availableMB‚ 738.88671875 proc.cpu.threads‚17) disk./.usagePercent‚76.00459807122148 disk.nvme0n1.in‚ 0.00390625 disk.md0.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625% disk./.usageGB‚1166.8605918884277 disk.nvme1n1.in‚0dŠLÞ“•‚(:ˆ ‡ÿõÒ©¼øË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.068' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ìœàb–‚(:× –ÿõÒ¬Ë) disk./.usagePercent‚76.00463409995318 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚30353.79150390625 disk.nvme0n1.in‚ 0.00390625' proc.memory.availableMB‚ 738.8828125 network.sent‚ 9.777751e+06% disk./.usageGB‚1166.8611450195312 disk.md0.in‚0" disk.dm-1.out‚8691.72509765625 disk.nvme1n1.in‚0" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836 disk.md0.out‚ 170.33203125! disk.dm-0.out‚40920.294921875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚19622.2060546875# memory_percent‚95.4902172088623 cpu‚25.63481995138107 network.recv‚ 1.486674e+06 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚30353.79150390625% disk.nvme1n1.out‚19622.2060546875 proc.cpu.threads‚17Ó;Óž—‚(:“ –ÿõÒÓ‡ôË gpu.0.gpu‚91.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚270.002' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.00066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ty—2˜- _wandbruntime‚5580  _runtime‚5580íZ1@Æ™À _runtime‚5580.842551972 _step‚507 & lr-AdamW/pg1‚0.0004910504727022978 & lr-AdamW/pg2‚0.0004910504727022978 trainer/global_step‚6199 $ _timestamp‚1.784512407266959e+09ü„>˚Š& lr-AdamW/pg1‚0.0004910504727022978 & lr-AdamW/pg2‚0.0004910504727022978 trainer/global_step‚6199 $ _timestamp‚1.784512407266959e+09 _runtime‚5580.842551972 _step‚507û)àÝ2›- _wandbruntime‚5580  _runtime‚5580)R í'œçO _step‚508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50162625 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25650713 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚955669080773120 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61316527 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚206184584404480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10818156 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213421870670848 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚948472663513600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚950109740151473 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25650427 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7196075835392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚948472646474240 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61316527 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50162625 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61326670 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7196075835392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111479150 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚948472646474240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7196417259520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚942924524544976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14832274 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚950110097752107 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7185215606497 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚206184603016704 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚955668722309632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50162623 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚942924541584336 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7185556167771 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14832557 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚213421998746624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚955669080773120 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7237286266368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111489295 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61326670 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚6199 % _timestamp‚1.7845124072679892e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚5580.843461955 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7237286266368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111489295 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61326670 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚6199 % _timestamp‚1.7845124072679892e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚5580.843461955 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50162625 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25650713 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚955669080773120 WJDeviceStatsMonitor.on_train_®‚ÙÄk batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61316527 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚206184584404480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10818156 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213421870670848 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚948472663513600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚950109740151473 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25650427 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7196075835392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚948472646474240 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61316527 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50162625 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61326670 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7196075835392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111479150 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚948472646474240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7196417259520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚942924524544976 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14832274 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚950110097752107 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7185215606497 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚206184603016704 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚955668722309632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50162623 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚942924541584336 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7185556167771 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14832557 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚213421998746624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚955669080773120 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7238405826048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚955818465381376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111506758 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25654346 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10819887 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7186323994749 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50170416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚948620920976384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5581.467163009 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚955818465381376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7197527365632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚948620938015744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚948620920976384 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111496599 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7197185934336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚509 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272=¤ü& öM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5581.467163009 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚955818465381376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7197527365632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚948620938015744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚948620920976384 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111496599 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7197185934336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚509 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111506758 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50170418 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61326183 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50170416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7238405826048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚955818465381376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111506758 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25654346 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10819887 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7186323994749 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50170416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚948620920976384ýF_Ù2¡- _wandbruntime‚5581  _runtime‚5581Š]D¢ÿ % _timestamp‚1.7845124078924596e+09 _runtime‚5581.467718031 _step‚510 .loss/FRt1/assignment_loss‚1.70431301459541 %loss/total_loss‚7.893305829066916 epoch‚15 0loss/FRt2/assignment_loss‚0.7572837365183601 2loss/FBt2/assignment_loss‚5.07386129520896e-05 /loss/FRt1/detection_loss‚0.4849874452036416 -loss/FRt2/detection_loss‚1.61894016571836 1loss/SRqqt1/detection_loss‚0.6661406708714526 trainer/global_step‚6199 2loss/SRqqt2/assignment_loss‚0.5556242906064829 /loss/FBt1/detection_loss‚0.5134215086996555 /loss/FBt2/detection_loss‚0.4382210495520611 3loss/SRqqt1/assignment_loss‚0.48781861844317764 2loss/FBt1/assignment_loss‚0.002635443846112821 /loss/SRqqt2/detection_loss‚0.66386914639925ª9lq £„ epoch‚15 0loss/FRt2/assignment_loss‚0.7572837365183601 2loss/FBt2/assignment_loss‚5.07386129520896e-05 /loss/FRt1/detection_loss‚0.4849874452036416 -loss/FRt2/detection_loss‚1.61894016571836 1loss/SRqqt1/detection_loss‚0.6661406708714526 trainer/global_step‚6199 2loss/SRqqt2/assignment_loss‚0.5556242906064829 /loss/FBt1/detection_loss‚0.5134215086996555 /loss/FBt2/detection_loss‚0.4382210495520611 3loss/SRqqt1/assignment_loss‚0.48781861844317764 2loss/FBt1/assignment_loss‚0.002635443846112821 /loss/SRqqt2/detection_loss‚0.66386914639925 % _timestamp‚1.7845124078924596e+09 _runtime‚5581.467718031 _step‚510 .loss/FRt1/assignment_loss‚1.70431301459541 %loss/total_loss‚7.893305829066916þqòÏL_¤‚(:Ô ¥ÿõÒÝŽÁË& disk.nvme0n1.out‚30355.12841796875 disk.md0.in‚0' proc.memory.availableMB‚ 738.8515625* proc.memory.percent‚18.392324447631836 cpu‚25.636106438512957( disk.nvme0c0n1.out‚30355.12841796875 disk.nvme0n1.in‚ 0.00390625# disk.nvme1n1.out‚19623.19140625" disk.dm-1.out‚8691.83447265625 network.sent‚ 9.780715e+06 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚19623.19140625 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0$ memory_percent‚95.49040794372559" proc.memory.rssMB‚ 3013.3984375 network.recv‚ 1.490314e+06) disk./.usagePercent‚76.00466391683459 disk.rbd0.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 170.7734375! disk.dm-0.out‚40921.552734375 proc.cpu.threads‚17% disk./.usageGB‚1166.8616027832031 disk.nvme0c0n1.in‚ 0.00390625ô[Œ¡ž¥‚(:“ ¥ÿõÒ²µÌ gpu.0.gpu‚85.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚202.333' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.44433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜à›Z¦‚(:Ï ´ÿõÒœ·Ë network.sent‚ 9.845016e+06% disk./.usageGB‚1166.8614616394043# disk.dm-0.out‚40926.48974609375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚30360.44287109375 disk.nvme0n1.in‚ 0.00390625" disk.nvme1n1.out‚ 19624.9921875" proc.memory.rssMB‚ 3013.3984375 disk.dm-1.out‚ 8692.078125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚95.49038410186768* proc.memory.percent‚18.392324447631836 network.recv‚ 1.495307e+06 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚76.00465472329616 disk.md0.in‚0 disk.md0.out‚ 171.703125 disk.dm-0.in‚ 0.00390625& disk.nvme0n1.out‚30360.44287109375$ disk.nvme1c1n1.out‚ 19624.9921875( proc.memory.availableMB‚ 738.85546875 cpu‚25.637077369625416|á4Œ“§‚(:ˆ ´ÿõÒ¶ßùË gpu.0.gpu‚58.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.505' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.835 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cʯ¬2¨- _wandbruntime‚5624  _runtime‚5624<©m¸©Á trainer/global_step‚6249 % _timestamp‚1.7845124514179065e+09 _runtime‚5624.995016939 _step‚511 & lr-AdamW/pg1‚0.0004890714734285846 & lr-AdamW/pg2‚0.0004º¢Ð890714734285846ÀŸ^ÅÌªÆ _runtime‚5624.995016939 _step‚511 & lr-AdamW/pg1‚0.0004890714734285846 & lr-AdamW/pg2‚0.0004890714734285846 trainer/global_step‚6249 % _timestamp‚1.7845124514179065e+09ÿ7½6æ2«- _wandbruntime‚5624  _runtime‚5624á•dí'¬çO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚955886388620800 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61801179 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚950296074957936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚215136863113216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10904853 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50552273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10904856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚957536756738481 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14946789 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚963138016518656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50552275 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207843597908480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚950296057918576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7251644937216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61811322 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7293137128960 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215136735037440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7293246592512 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚5624.995080569 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚955886371581440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚6249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚957537114339115 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61811322 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚955886388620800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 % _timestamp‚1.7845124514185698e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112353452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _step‚512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚963138374982144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7251644937216 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚955886371581440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚207843616520704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7241039381179 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50552275 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚305 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14946487 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7241039381179 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50552275 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚305 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14946487 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14946789 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚963138016518656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50552275 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207843597908480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚950296057918576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7251644937216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61811322 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7293137128960 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215136735037440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7293246592512 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚5624.995080569 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚955886371581440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚6249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚957537114339115 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61811322 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚955886388620800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 % _timestamp‚1.7845124514185698e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112353452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _step‚512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184€:¨™Pb®‚(:× ÃÿõÒ“»ÒË disk.md0.in‚0 disk.rbd0.in‚0# memory_percent‚95.4918622970581* proc.memory.percent‚18.392324447631836 cpu‚25.638394859265617 proc.cpu.threads‚17 network.sent‚ 9.860781e+06% disk./.usageGB‚1166.8618431091309 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚19625.92529296875 disk.md0.out‚ 172.22265625 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚40927.37255859375 disk.rbd0.out‚0 network.recv‚ 1.498544e+06) disk./.usagePercent‚76.00467957069733& disk.nvme1n1.out‚19625.92529296875 disk.dm-1.in‚0 disk.dm-1.out‚ 8692.1171875 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚30361.5400390625% disk.nvme0n1.out‚30361.5400390625 disk.nvme0n1.in‚ 0.00390625( proc.memory.availableMB‚ 738.61328125" proc.memory.rssMB‚ 3013.3984375 disk.nvme1n1.in‚0Ó<³œ¯‚(:‘ ÃÿõÒ§—Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚165.4' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚55.13333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðp—‡2°- _wandbruntime‚5625  _runtime‚5625—T!-÷&±ñM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61810835 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚957685653394951 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112381060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7294356363776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50560066 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215171027183104 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50560068 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112370901 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _step‚513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7253096467456 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚957685295794205 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚963287759590400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7241807208157 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25855277 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7252755036160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112370901 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112381060 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50560068 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚956034646083584 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7253096467456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50560066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7294249004544 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚6249 _runtime‚5625.616640394 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845124520408459e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚963287759590400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚956034663122944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7242147769543 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚302 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7252755036160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10906587 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61810835 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14948992Ä S§²öM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10906590 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25855277 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 V7=ÔhU$HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7252755036160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112370901 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112381060 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50560068 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚956034646083584 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7253096467456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50560066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7294249004544 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚6249 _runtime‚5625.616640394 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845124520408459e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚963287759590400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚956034663122944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7242147769543 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚302 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7252755036160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10906587 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61810835 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14948992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61810835 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚957685653394951 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112381060 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7294356363776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50560066 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215171027183104 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50560068 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112370901 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _step‚513 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7253096467456 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚957685295794205 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚963287759590400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7241807208157 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚957667591292176 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚970607310727680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚963300113728000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10991553 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50941923 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚963300113728000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209502611412480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7349037809664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7307555463168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 _runtime‚5669.255048274 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚6299 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚963300096688640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62295974 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50941925 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚970607310727680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚963300113728000 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10991553 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50941923 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚963300113728000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209502611412480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7349037809664 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7307555463168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 _runtime‚5669.255048274 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚6299 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚963300096688640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62295974 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50941925 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚957667591292176 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26056540 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _runtime‚5669.873679888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚963448371190784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15063250 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚965112312381213 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26056254 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7350152455168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚963448371190784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113255362 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50949718 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62305644 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7308665569280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113255362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7297290421565 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62295487 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚957815021959648 trainer/global_step‚6299Xùü&ÂöM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26056540 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _runtime‚5669.873679888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚963448371190784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15063250 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚965112312381213 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26056254 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7350152455168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚963448371190784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113255362 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50949718 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62305644 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7308665569280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113255362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7297290421565 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62295487 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚957815021959648 trainer/global_step‚6299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚970757053799424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚517 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚963448388230144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10993287 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚965112669981959 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7297630982951 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚963448388230144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62305644 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113245203 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7308324137984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7308665569280 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62295487 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50949716 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50949716 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216885836778496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184… ®Æ2Ã- _wandbruntime‚5669  _runtime‚5669‘™ÈÛ Ä„ 2loss/FBt2/assignment_loss‚0.012604850866731354 /loss/FRt2/detection_loss‚1.5337882596655705 1loss/FBt1/assignment_loss‚0.03131525808379235 epoch‚15 _runtime‚5669.874179159 2loss/SRqqt1/assignment_loss‚0.4982452840798924 /loss/FRt1/detection_loss‚0.5422835160690738 1loss/SRqqt2/detection_loss‚0.6884393031355894 /loss/FBt2/detection_loss‚0.5998665016906505 &loss/total_loss‚7.9037047351393515 0loss/FRt2/assignment_loss‚0.7450542401699792 2loss/SRqqt2/assignment_loss‚0.5186590191564003 1loss/SRqqt1/detection_loss‚0.6756173736468434 /loss/FBt1/detection_loss‚0.4468447365793613 trainer/global_step‚6299 % _timestamp‚1.7845124962989292e+09 _step‚518 0loss/FRt1/assignment_loss‚1.6109863919954674¸¢Å‰ % _timestamp‚1.7845124962989292e+09 _step‚518 0loss/FRt1/assignment_loss‚1.6109863919954674 2loss/FBt2/assignment_loss‚0.012604850866731354 /loss/FRt2/detection_loss‚1.5337882596655705 1loss/FBt1/assignment_loss‚0.03131525808379235 epoch‚15 _runtime‚5669.874179159 2loss/SRqqt1/assignment_loss‚0.4982452840798924 /loss/FRt1/detection_loss‚0.5422835160690738 1loss/SRqqt2/detection_loss‚0.6884393031355894 /loss/FBt2/detection_loss‚0.5998665016906505 &loss/total_loss‚7.9037047351393515 0loss/FRt2/assignment_loss‚0.7450542401699792 2loss/SRqqt2/assignment_loss‚0.5186590191564003 1loss/SRqqt1/detection_loss‚0.6756173736468434 /loss/FBt1/detection_loss‚0.4468447365793613 trainer/global_step‚6299†À|þÃaÆ‚(:Ö ðÿõÒ´™äË" proc.memory.rssMB‚ 3013.3984375& disk.nvme0n1.out‚30369.61962890625* proc.memory.percent‚18.392324447631836 cpu‚25.641519045338267 proc.cpu.threads‚17& disk.nvme1n1.out‚19627.92333984375% disk./.usageGB‚1166.6241645812988 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme1c1n1.out‚19627.92333984375 disk.md0.out‚ 173.27734375 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚40934.8759765625 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚30369.61962890625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 8692.4296875' proc.memory.availableMB‚ 492.4609375 network.sent‚ 9.969431e+06 network.recv‚ 1.511326e+06( disk./.usagePercent‚75.9891981489203 disk.nvme0n1.in‚ 0.00390625# memory_percent‚96.9942569732666aužÇ‚(:“ ðÿõÒ£§Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚284.347' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.78233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|-rs\È‚(:Ñ ÿÿõÒµ„ÊË" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836 disk.nvme1n1.in‚0& disk.nvme1n1.out‚19628.52099609375 disk.md0.out‚ 173.6953125 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚30370.52587890625 network.sent‚ 1.0020942e+07) disk./.usagePercent‚75.98920386382257 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 492.46484375 network.recv‚ 1.516665e+06$ disk./.usageGB‚1166.624252319336 disk.md0.in‚0 disk.dm-1.out‚8692.5& disk.nvme0n1.out‚30370.52587890625 cpu‚25.64242029749783 proc.cpu.threads‚17" disk.dm-0.out‚40935.4267578125 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme1c1n1.out‚19628.52099609375$ memory_percent‚96.994233131408693BV žÉ‚(:“ ÿÿõÒŸ¸‹Ì gpu.0.gpu‚55.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.453' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.81766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jƒÇÃcÊ‚(:Ø Ž€öÒÜýÇË network.sent‚ 1.0025849e+07 network.recv‚ 1.519922e+06 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚96.99420928955078% proc.memory.availableMB‚ 492.46875* proc.memory.percent‚18.392324447631836) disk./.usagePercent‚75.98918050726546 disk.md0.out‚ 174.2109375& disk.nvme0n1.out‚30386.56103515625( disk.nvme1c1n1.out‚19629.70166015625 cpu‚25.643664930359275 proc.cpu.threads‚17$ disk./.usageGB‚1166.623893737793 disk.md0.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625& disk.nvme1n1.out‚19629.70166015625" disk.dm-0.out‚40951.3916015625! disk.dm-1.out‚8692.6494140625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚30386.56103515625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0" proc.memory.rssMB‚ 3013.3984375‚‘»ÈË‚(:’ Ž€öÒ–ãˆÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.246' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.748666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w^§22Ì- _wandbruntime‚5712  _runtime‚5712Žés¡ÉÍà ' lr-AdamW/pg1‚0.00048507856755911365 ' lr-AdamW/pg2‚0.00048507856755911365 trainer/global_step‚6349 % _timestamp‚1.7845125393966768e+09 _runtime‚5712.973091423 _step‚519ܦÎÎÈ ' lr-AdamW/pg1‚0.00048507856755911365 ' lr-AdamW/pg2‚0.00048507856755911365 trainer/global_step‚6349 % _timestamp‚1.7845125393966768e+09 _runtime‚5712.973091423 _step‚519‡)>x2Ï- _wandbruntime‚5712  _runtime‚5712‡¡I SÐçO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚970713821795840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚978076963400192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7362783140864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26253363 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114102056 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚970713821795840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7363124564992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7352005807995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5712.973184113 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚970713838835200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218566399424000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211161624916480 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚965039141705136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51331573 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚978076604936704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7404774507520 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15174804 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚978076604936704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7404886068224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184i]Ç2ò'ÑìO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15174804 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚978076604936704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7404886068224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚970713821795840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚978076963400192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7362783140864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26253363 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114102056 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚970713821795840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7363124564992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7352005807995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5712.973184113 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚970713838835200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218566399424000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211161624916480 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚965039141705136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51331573 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚520 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚978076604936704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7404774507520 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232ˆ, ­Œ2Ò- _wandbruntime‚5713  _runtime‚5713®ç¸÷&ÓñM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7364234671104 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7405995962368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚965186555333248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7364234671104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62780139 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚6349 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51339368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26257334 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15177067 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚978226348008448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚970862096297984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26257054 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218600700081152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7363893239808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚972539686568967 % _timestamp‚1.7845125400217931e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62790296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51339366 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚970862096297984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7405894894592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114119505 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51339368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11079990 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7353114196359 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114129664 _step‚521 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114129664 _step‚521 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚965186555333248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7364234671104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62780139 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚6349 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51339368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26257334 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15177067 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚978226348008448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚970862096297984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26257054 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218600700081152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7363893239808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_byte[ˆ{Ô s.all.allocated‚972539686568967 % _timestamp‚1.7845125400217931e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62790296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51339366 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚970862096297984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7405894894592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114119505 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51339368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11079990 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7353114196359‰×ËÇ–2Õ- _wandbruntime‚5713  _runtime‚5713§¯~ Ö‡ 0loss/FRt1/assignment_loss‚1.5898631090247952 2loss/SRqqt1/assignment_loss‚0.4939849669978366 2loss/SRqqt2/assignment_loss‚0.5344807970935341 1loss/SRqqt1/detection_loss‚0.7249353131858787 1loss/SRqqt2/detection_loss‚0.6771980028296897 &loss/total_loss‚7.8882842813834735 epoch‚15 % _timestamp‚1.7845125400228715e+09 0loss/FRt2/assignment_loss‚0.7880091767679612 /loss/FRt1/detection_loss‚0.4833700477194346 /loss/FRt2/detection_loss‚1.5488916930984662 0loss/FBt1/detection_loss‚0.39619198641940173 _runtime‚5713.598136858 _step‚522 2loss/FBt1/assignment_loss‚0.001444810396482806 3loss/FBt2/assignment_loss‚0.0015941626469731145 /loss/FBt2/detection_loss‚0.6483202152030201 trainer/global_step‚63498ÕÁN׌ 2loss/FBt1/assignment_loss‚0.001444810396482806 3loss/FBt2/assignment_loss‚0.0015941626469731145 /loss/FBt2/detection_loss‚0.6483202152030201 trainer/global_step‚6349 0loss/FRt1/assignment_loss‚1.5898631090247952 2loss/SRqqt1/assignment_loss‚0.4939849669978366 2loss/SRqqt2/assignment_loss‚0.5344807970935341 1loss/SRqqt1/detection_loss‚0.7249353131858787 1loss/SRqqt2/detection_loss‚0.6771980028296897 &loss/total_loss‚7.8882842813834735 epoch‚15 % _timestamp‚1.7845125400228715e+09 0loss/FRt2/assignment_loss‚0.7880091767679612 /loss/FRt1/detection_loss‚0.4833700477194346 /loss/FRt2/detection_loss‚1.5488916930984662 0loss/FBt1/detection_loss‚0.39619198641940173 _runtime‚5713.598136858 _step‚522Šã0aØ‚(:Ö €öÒŸ’ðË% disk./.usageGB‚1166.8617362976074 disk.rbd0.in‚0% disk.nvme0n1.out‚30607.2353515625 proc.cpu.threads‚17 network.recv‚ 1.523568e+06' disk./.usagePercent‚76.004672613425' disk.nvme0c0n1.out‚30607.2353515625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 174.63671875 disk.dm-0.in‚ 0.00390625 network.sent‚ 1.0041614e+07 disk.nvme1c1n1.in‚0! disk.dm-1.out‚8692.7392578125$ memory_percent‚96.99466228485107( proc.memory.availableMB‚ 492.39453125" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836 cpu‚25.64463822747132 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚19653.4619140625% disk.nvme1n1.out‚19653.4619140625" disk.dm-0.out‚41194.8330078125 disk.dm-1.in‚0‚ ªÜžÙ‚(:“ €öÒû²Ì gpu.0.gpu‚47.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚289.222' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.40733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÓ'ëaÚ‚(:Ö ¬€öÒ‹¬¥Ë cpu‚25.645571339514774 network.sent‚ 1.0090532e+07% disk./.usageGB‚1166.6245803833008 disk.dm-1.out‚ 8692.8828125 disk.dm-0.in‚ 0.00390625" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836 network.recv‚ 1.52803e+06 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚30614.47802734375( disk.nvme1c1n1.out‚19654.15185546875& disk.nvme1n1.out‚19654.15185546875 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚30614.47802734375 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 174.97265625$ memory_percent‚96.99468612670898& proc.memory.availableMB‚ 492.390625) disk./.usagePercent‚75.98922523258757 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0! disk.dm-0.out‚41201.888671875ƒÙXžÛ‚(:“ ¬€öÒ¨âøË gpu.0.gpu‚58.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚242.836' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚80.94533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˱ÒQbÜ‚(:× »€öÒŽŠ°Ë( disk.nvme0c0n1.out‚30615.32177734375& disk.nvme0n1.out‚30615.32177734375 disk.nvme1n1.in‚0 network.sent‚ 1.0097226e+07! disk.dm-0.out‚41202.541015625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚19654.55810546875 cpu‚25.646580124104243 network.recv‚ 1.531499e+06) disk./.usagePercent‚75.98919566418017% disk./.usageGB‚1166.6241264343262 disk.nvme0n1.in‚ 0.00390625# memory_percent‚96.9947338104248' proc.memory.availableMB‚ 492.3828125 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚ 175.2109375 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme1n1.out‚19654.55810546875" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836 disk.dm-1.in‚0 disk.dm-1.out‚ 8692.94140625 disk.rbd0.in‚0œ¿+*žÝ‚(:“ »€öÒ€¨ðË gpu.0.gpu‚41.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚284.258' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.75266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B?¬B2Þ- _wandbruntime‚5757  _runtime‚57572CkÆßÀ & lr-AdamW/pg2‚0.0004830649174202386 trainer/global_step‚6399 $ _timestamp‚1.784512583511011e+09 _runtime‚5757.086607901 _step‚523 & lr-AdamW/pg1‚0.0004830649174202386œ üËàÅ & lr-AdamW/pg1‚0.0004830649174202386 & lr-AdamW/pg2‚0.0004830649174202386 trainer/global_step‚6399 $ _timestamp‚1.784512583511011e+09 _runtime‚5757.086607901 _step‚523‹añ‹2á- _wandbruntime‚5757  _runtime‚57573¶·Ýê'âäO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7418693666816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11164953 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7407489021403 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51721223 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26453871 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784512583512066e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚978127546903040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7460603783168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚972410675078736 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚972410658039376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚978127546903040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51721223 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚985545899145728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 trainer/global_step‚6399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚985546257609216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _step‚524 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15288918 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220281357696512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚985546257609216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚978127563942400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51721225 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚978127563942400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114976358 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212820657032704 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚985546257609216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚978127563942400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51721225 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚978127563942400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114976358 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212820657032704 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚985545899145728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 trainer/global_step‚6399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚985546257609216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _step‚524 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180Œýh;T2ä- _wandbruntime‚5757  _runtime‚5757’‘_ åñM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7408256848381 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115003966 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚978275821405184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114993807 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63264791 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212853837302784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212853818690560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7461819640832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚979966703155975 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51729018 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115003966 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚985695642217472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7419462341632 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11166687 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63264791 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51729016 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26457839 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7461712281600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚978275838444544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚979966345555229 _runtime‚5757.710011152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51729018 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15291152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚6399 _step‚525 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚972558105746208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚985695642217472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7419803772928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63274948 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚978275821405184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26458105 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114993807 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚985695283746816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26457839 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7461712281600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚978275838444544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚979966345555229 _runtime‚5757.710011152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51729018 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15291152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚6399 _step‚525 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚972558105746208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚985695642217472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7419803772928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63274948 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚978275821405184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26458105 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114993807 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚985695283746816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115003966 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚978275821405184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114993807 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63264791 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212853837302784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚212853818690560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7461819640832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚979966703155975 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51729018 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0‰¢2ç- _wandbruntime‚5757  _runtime‚5757³m¾ïè€ 0loss/SRqqt2/assignment_loss‚0.55119042607711 /loss/FRt1/detection_loss‚0.4584414724922963 .loss/FRt2/detection_loss‚1.500537364006259 1loss/SRqqt1/detection_loss‚0.7348161990422523 /loss/FBt1/detection_loss‚0.5777081299296486 _step‚526 2loss/SRqqt1/assignment_loss‚0.5130814482225197 1loss/FBt1/assignment_loss‚0.06940872932970928 3loss/FBt2/assignment_loss‚0.0005636417161185358 1loss/SRqqt2/detection_loss‚0.6463352797679673 /loss/FBt2/detection_loss‚0.4320337114013022 epoch‚15 trainer/global_step‚6399 $ _timestamp‚1.784512584135301e+09 0loss/FRt1/assignment_loss‚1.6234839811906694 0loss/FRt2/assignment_loss‚0.7318149786307218 %loss/total_loss‚7.839415361806575 _runtime‚5757.710559683ƒºgè é… epoch‚15 trainer/global_step‚6399 $ _timestamp‚1.784512584135301e+09 0loss/FRt1/assignment_loss‚1.6234839811906694 0loss/FRt2/assignment_loss‚0.7318149786307218 %loss/total_loss‚7.839415361806575 _runtime‚5757.710559683 0loss/SRqqt2/assignment_loss‚0.55119042607711 /loss/FRt1/detection_loss‚0.4584414724922963 .loss/FRt2/detection_loss‚1.500537364006259 1loss/SRqqt1/detection_loss‚0.7348161990422523 /loss/FBt1/detection_loss‚0.5777081299296486 _step‚526 2loss/SRqqt1/assignment_loss‚0.5130814482225197 1loss/FBt1/assignment_loss‚0.06940872932970928 3loss/FBt2/assignment_loss‚0.0005636417161185358 1loss/SRqqt2/detection_loss‚0.6463352797679673 /loss/FBt2/detection_loss‚0.4320337114013022Žž`ê‚(:Õ Ê€öÒîÎË% disk./.usageGB‚1166.6241264343262 disk.dm-1.in‚0" disk.dm-1.out‚8693.01513671875' disk.nvme0c0n1.out‚30636.6884765625 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚19656.92919921875 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 network.sent‚ 1.0112991e+07) disk./.usagePercent‚75.98919566418017 disk.md0.out‚ 175.54296875$ memory_percent‚96.99478149414062# proc.memory.availableMB‚492.375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme0n1.out‚30636.6884765625 disk.nvme1n1.in‚0& disk.nvme1n1.out‚19656.92919921875" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836 network.recv‚ 1.534659e+06 disk.nvme1c1n1.in‚0" disk.dm-0.out‚41225.4775390625 cpu‚25.6486643141454 proc.cpu.threads‚17K†ò žë‚(:“ Ê€öÒª¥‘Ì gpu.0.gpu‚79.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚125.581' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.86033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lzŽ2ì- _wandbruntime‚5774  _runtime‚5774:Aà˜º3í¨z 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.281589150428772 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7467373609542847 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7639179229736328 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4084327220916748 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40500855445861816 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15546058118343353 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976779222488403 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24488131701946259 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6898906826972961 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9974282383918762 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6204013824462891 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22369030117988586 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2942137122154236 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029341697692871094 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.622269332408905 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9969152808189392 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8442283868789673 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5748046040534973 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03900497406721115 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_4‚0.09102103859186172 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7377346754074097 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21623751521110535 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.579134464263916 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0476805716753006 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05151824280619621 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.40037256479263306 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2810502350330353 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9064008593559265 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9967930316925049 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25063174962997437 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3339826762676239 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6720782518386841 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 *jet/accuracy_2_of_3‚0.5970236659049988 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7647772431373596 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7113539576530457 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7836425304412842 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6635266542434692 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992209672927856 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.751134991645813 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7391517758369446 particle/accuracy_2_of_4‚1 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6640625 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24867673218250275 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6853129267692566 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7887786030769348 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3379497230052948 )particle/sensitivity‚0.64818274974823 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7899882197380066 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8796780109405518 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7712899446487427 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7882653474807739 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8585602045059204 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_1_of_1‚0.361611545085907 particle/accuracy_3_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6501942873001099 % _timestamp‚1.7845126010115054e+09 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.862622857093811 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.031741831451654434 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9917343854904175 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7067212462425232 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7460612058639526 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02383851632475853 *jet/accuracy_1_of_2‚0.8109635710716248 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4832455813884735 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029119333252310753 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2125910371541977 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29257234930992126 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8658164143562317 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.36109980940818787 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6853129267692566 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27408894896507263 particle/accuracy_2_of_2‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 jet/accuracy_1_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7495810985565186 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2402094453573227 particle/accuracy_1_of_1‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3906250298023224 epoch‚15 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2726527154445648 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5153977274894714 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6502488851547241 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2930014133453369 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7290722727775574 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8717687726020813 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6946467757225037 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23340889811515808 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.7871630787849426 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8915415406227112 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14431217312812805 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6799761652946472 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0740424171090126 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 jet/accuracy_5_of_5‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03827359154820442 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4923335015773773 )loss/val_total_loss‚7.823483658303797 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9539677500724792 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17134591937065125 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24953903257846832 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.880536675453186 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29282593727111816 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 _step‚527 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48869651556015015 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 particle/accuracy_3_of_3‚1 trainer/global_step‚6399 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7226978540420532 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09242232888936996 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7088301777839661 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6816765069961548 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6604312062263489 7"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.78899747133255 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 _runtime‚5774.587586978 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 +jet/accuracy_2_of_2‚0.16987711191177368 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02383851632475853 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9911690950393677 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03216278925538063 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +particle/specificity‚0.9147059917449951 particle/accuracy_1_of_5‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7380256652832031 'particle/f_score‚0.6613783836364746 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5595673322677612 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6573617458343506 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2229267954826355 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7192782163619995 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23906934261322021 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9338030815124512 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9971752166748047 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5562499761581421 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23651167750358582 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.84422838687øfB&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6908587217330933 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988561272621155 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5434274673461914 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7391536831855774 particle/accuracy_3_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.281589150428772 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15977604687213898 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 (particle/accuracy‚0.8574645519256592 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7638770341873169 /particle/accuracy_4_of_4‚0.9558894634246826 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_3_of_3‚0.07873178273439407 *jet/accuracy_1_of_4‚0.9774220585823059 particle/accuracy_5_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32621756196022034 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108°¥Ù3=î­z 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25063174962997437 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3339826762676239 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6720782518386841 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 *jet/accuracy_2_of_3‚0.5970236659049988 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7647772431373596 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7113539576530457 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7836425304412842 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6635266542434692 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992209672927856 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.751134991645813 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7391517758369446 particle/accuracy_2_of_4‚1 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6640625 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24867673218250275 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6853129267692566 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7887786030769348 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3379497230052948 )particle/sensitivity‚0.64818274974823 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7899882197380066 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8796780109405518 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7712899446487427 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7882653474807739 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8585602045059204 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_1_of_1‚0.361611545085907 particle/accuracy_3_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6501942873001099 % _timestamp‚1.7845126010115054e+09 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.862622857093811 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.031741831451654434 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9917343854904175 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7067212462425232 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7460612058639526 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02383851632475853 *jet/accuracy_1_of_2‚0.8109635710716248 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4832455813884735 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029119333252310753 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2125910371541977 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29257234930992126 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8658164143562317 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.36109980940818787 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6853129267692566 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27408894896507263 particle/accuracy_2_of_2‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 jet/accuracy_1_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7495810985565186 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2402094453573227 particle/accuracy_1_of_1‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3906250298023224 epoch‚15 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2726527154445648 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5153977274894714 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6502488851547241 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2930014133453369 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7290722727775574 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8717687726020813 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6946467757225037 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23340889811515808 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.7871630787849426 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8915415406227112 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14431217312812805 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6799761652946472 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0740424171090126 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 jet/accuracy_5_of_5‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03827359154820442 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4923335015773773 )loss/val_total_loss‚7.823483658303797 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9539677500724792 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17134591937065125 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24953903257846832 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.880536675453186 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29282593727111816 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 _step‚527 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48869651556015015 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 particle/accuracy_3_of_3‚1 trainer/global_step‚6399 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7226978540420532 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09242232888936996 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7088301777839661 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6816765069961548 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6604312062263489 7"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.78899747133255 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 _runtime‚5774.587586978 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 +jet/accuracy_2_of_2‚0.16987711191177368 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02383851632475853 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9911690950393677 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03216278925538063 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +particle/specificity‚0.9147059917449951 particle/accuracy_1_of_5‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7380256652832031 'particle/f_score‚0.6613783836364746 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5595673322677612 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6573617458343506 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2229267954826355 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7192782163619995 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23906934261322021 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9338030815124512 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9971752166748047 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5562499761581421 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23651167750358582 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8442283868789673 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6908587217330933 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988561272621155 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5434274673461914 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7391536831855774 particle/accuracy_3_of_4‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.281589150428772 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15977604687213898 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 (particle/accuracy‚0.8574645519256592 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7638770341873169 /particle/accuracy_4_of_4‚0.9558894634246826 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_3_of_3‚0.07873178273439407 *jet/accuracy_1_of_4‚0.9774220585823059 particle/accuracy_5_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32621756196022034 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.281589150428772 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7467373609542847 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7639179229736328 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4084327220916748 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40500855445861816 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15546058118343353 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976779222488403 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24488131701946259 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6898906826972961 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9974282383918762 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6204013824462891 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22369030117988586 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2942137122154236 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029341697692871094 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.622269332408905 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9969152808189392 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8442283868789673 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5748046040534973 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03900497406721115 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_4‚0.09102103859186172 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7377346754074097 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21623751521110535 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.579134464263916 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0476805716753006 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05151824280619621 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.40037256479263306 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2810502350330353 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9064008593559265 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9967930316925049n™”ffï‚(:Û Ù€öÒ‘‚ñË disk.md0.out‚ 175.89453125 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625* proc.memory.percent‚18.392324447631836 cpu‚25.65552908302206 proc.cpu.threads‚17 network.recv‚ 1.540107e+06" disk.dm-0.out‚41226.1337890625( proc.memory.availableMB‚ 492.36328125" proc.memory.rssMB‚ 3013.3984375) disk./.usagePercent‚75.98919491875814% disk./.usageGB‚1166.6241149902344 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0" disk.dm-1.out‚8693.09716796875( disk.nvme0c0n1.out‚30637.62939453125 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚19657.4755859375 network.sent‚ 1.0175448e+07 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚30637.62939453125 disk.nvme1n1.in‚0% disk.nvme1n1.out‚19657.4755859375$ memory_percent‚96.99485301971436ÂÒØWœð‚(:‘ Ù€öÒú¸Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚53.0 gpu.0.powerWatts‚117.988' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.32933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É¿úsdñ‚(:Ù è€öÒ’¸ÆË cpu‚25.65547481717881! disk.dm-0.out‚41227.876953125) disk./.usagePercent‚75.98924312271642( disk.nvme1c1n1.out‚19903.42236328125 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.rbd0.in‚0 network.sent‚ 1.0219314e+07$ disk./.usageGB‚1166.624855041504! disk.dm-1.out‚8938.5146484375( disk.nvme0c0n1.out‚30639.49560546875$ memory_percent‚95.49365043640137' proc.memory.availableMB‚ 738.3203125" proc.memory.rssMB‚ 3013.3984375 network.recv‚ 1.545272e+06& disk.nvme0n1.out‚30639.49560546875 disk.md0.out‚ 176.19140625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚19903.42236328125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625* proc.memory.percent‚18.392324447631836¦v žò‚(:“ è€öÒñŒŽÌ gpu.0.gpu‚54.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚250.508' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚83.50266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TŽ«fó‚(:Û ÷€öÒ¢³áË& disk.nvme0n1.out‚30757.73583984375 disk.nvme1n1.in‚0 disk.md0.in‚0* proc.memory.percent‚18.392324447631836 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚30757.73583984375' proc.memory.availableMB‚ 738.3203125 cpu‚25.654943419558844 proc.cpu.threads‚17 network.recv‚ 1.548916e+06 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚20162.20556640625 disk.md0.out‚ 176.67578125 disk.dm-0.in‚ 0.00390625! disk.dm-1.out‚9184.0341796875$ memory_percent‚95.49365043640137 network.sent‚ 1.0226119e+07) disk./.usagePercent‚76.00672103317801 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚20162.20556640625! disk.dm-0.out‚41358.353515625 disk.dm-1.in‚0" proc.memory.rssMB‚ 3013.3984375% disk./.usageGB‚1166.8931846618652rÿIºô‚(:’ ÷€öÒ⇬Ì gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚278.41' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.80333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áh¯2õ- _wandbruntime‚5818  _runtime‚5818C†7DÅö¿ % lr-AdamW/pg1‚0.000481039974678048 % lr-AdamW/pg2‚0.000481039974678048 trainer/global_step‚6449 % _timestamp‚1.7845126448109443e+09 _runtime‚5818.386969223 _step‚528“³Ê÷Ä _step‚528 % lr-AdamW/pg1‚0.000481039974678048 % lr-AdamW/pg2‚0.000481039974678048 trainer/global_step‚6449 % _timestamp‚1.7845126448109443e+09 _runtime‚5818.386969223cœãp2ø- _wandbruntime‚5818  _runtime‚5818(ú7í'ùçO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚994978213532160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7482935522439 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚987483615885824 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚989186382943639 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845126448116446e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15440463 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚994977855068672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚222039524297728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63831429 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11255772 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63821286 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214502249199104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7537275098624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26696232 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚994977855068672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116054967 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52233683 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚981703464460560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63821286 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26696492 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7494580606976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52233681 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15440720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52233681 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚994978213532160jà'…úìO QCDevice#„°Ûâ'StatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚994978213532160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7482935522439 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚987483615885824 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚989186382943639 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845126448116446e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15440463 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚994977855068672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚222039524297728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63831429 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11255772 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63821286 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214502249199104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7537275098624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26696232 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚994977855068672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116054967 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52233683 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚981703464460560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63821286 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26696492 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7494580606976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52233681 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15440720 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52233681 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚994978213532160‘Çà(2û- _wandbruntime‚5819  _runtime‚5819wæ}Ûö&üðM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7484043910691 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116082575 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63841099 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚989334921999363 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15442955 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚222073697018368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚5819.010174312 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116082575 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚987631907427328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚6449 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52241474 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚995127598140416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214535410856960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚995127598140416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15442668 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52241476 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚987631890387968 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11257506 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 $ _timestamp‚1.784512645434438e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63830942 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚981850878088672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7484384472077 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚987631907427328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116072416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214535429469184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7495690713088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚987631890387968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚981850895128032 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116072416 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26700171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272Vܪ‡û&ýõM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚995127598140416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214535410856960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚995127598140416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15442668 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52241476 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚987631890387968 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11257506 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 $ _timestamp‚1.784512645434438e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63830942 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚981850878088672 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7484384472077 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚987631907427328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116072416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214535429469184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7495690713088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚987631890387968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚981850895128032 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116072416 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26700171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7484043910691 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116082575 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63841099 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚989334921999363 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15442955 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚222073697018368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚5819.010174312 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116082575 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚987631907427328 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚6449 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52241474’[y±÷2þ- _wandbruntime‚5819  _runtime‚5819€kœYÿ 0loss/SRqqt2/detection_loss‚0.722391260159919 trainer/global_step‚6449 _step‚531 0loss/FRt2/assignment_loss‚0.7023931468166591 2loss/SRqqt1/assignment_loss‚0.4943982473190803 2loss/SRqqt2/assignment_loss‚0.5453373462115013 1loss/SRqqt1/detection_loss‚0.8145863653990091 .loss/FBt2/detection_loss‚0.642238287535515 $ _timestamp‚1.784512645435442e+09 1loss/FBt2/assignment_loss‚0.05964168528323388 /loss/FRt2/detection_loss‚1.6610677347412774 0loss/FBt1/detection_loss‚0.40934071352794055 %loss/total_loss‚8.221412982335462 epoch‚16 _runtime‚5819.010696823 2loss/FBt1/assignment_loss‚0.008224348442619581 0loss/FRt1/assignment_loss‚1.6992591176631946 /loss/FRt1/detection_loss‚0.4625347292355114EGÀž €† 2loss/FBt1/assignment_loss‚0.008224348442619581 0loss/FRt1/assignment_loss‚1.6992591176631946 /loss/FRt1/detection_loss‚0.4625347292355114 0loss/SRqqt2/detection_loss‚0.722391260159919 trainer/global_step‚6449 _step‚531 0loss/FRt2/assignment_loss‚0.7023931468166591 2loss/SRqqt1/assignment_loss‚0.4943982473190803 2loss/SRqqt2/assignment_loss‚0.5453373462115013 1loss/SRqqt1/detection_loss‚0.8145863653990091 .loss/FBt2/detection_loss‚0.642238287535515 $ _timestamp‚1.784512645435442e+09 1loss/FBt2/assignment_loss‚0.05964168528323388 /loss/FRt2/detection_loss‚1.6610677347412774 0loss/FBt1/detection_loss‚0.40934071352794055 %loss/total_loss‚8.221412982335462 epoch‚16 _runtime‚5819.010696823“Τ\‚(:Ñ †öÒä«ïË disk.rbd0.out‚0" proc.memory.rssMB‚ 3013.3984375 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme1n1.in‚0 cpu‚25.65428715874255 proc.cpu.threads‚17# disk.dm-0.out‚41362.27685546875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625# disk.nvme0c0n1.out‚ 30761.921875! disk.nvme0n1.out‚ 30761.921875' disk.nvme1c1n1.out‚20163.0478515625% disk.nvme1n1.out‚20163.0478515625( proc.memory.availableMB‚ 738.27734375 network.sent‚ 1.0241884e+07) disk./.usagePercent‚76.00674563210517 disk.md0.out‚ 177.1328125 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0$ memory_percent‚95.49391269683838* proc.memory.percent‚18.392324447631836 network.recv‚ 1.552067e+06% disk./.usageGB‚1166.8935623168945 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚9184.16845703125‹Ó$P‚‚(:“ †öÒ¿ý¶ÌVyžƒ gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚273.317' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.10566666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dÁjqcƒ‚(:Ø •öÒ¢ÿÙË disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0c0n1.out‚30824.646484375 disk.nvme0n1.in‚ 0.00390625# disk.dm-0.out‚41476.43701171875" proc.memory.rssMB‚ 3013.3984375) disk./.usagePercent‚75.98926300063737$ disk.nvme0n1.out‚30824.646484375 disk.md0.out‚ 177.65234375$ memory_percent‚95.49405574798584( proc.memory.availableMB‚ 738.25390625* proc.memory.percent‚18.392324447631836 cpu‚25.65367955777807 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚20215.7529296875% disk.nvme1n1.out‚20215.7529296875 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚9184.33251953125 network.sent‚ 1.0293258e+07 network.recv‚ 1.557588e+06% disk./.usageGB‚1166.6251602172852-ê§Íž„‚(:“ •öÒÕá§Ì gpu.0.gpu‚48.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.655' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.55166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Rà˜d…‚(:Ù ¤öÒž¾¡Ë disk.rbd0.in‚0* proc.memory.percent‚18.392324447631836 cpu‚25.65367419509923 disk.md0.out‚ 177.89453125 network.sent‚ 1.0296281e+07 network.recv‚ 1.561216e+06 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚30852.76318359375 disk.dm-0.out‚41508.44921875 disk.dm-1.in‚0" disk.dm-1.out‚9186.15576171875) disk./.usagePercent‚75.97377188837386% disk./.usageGB‚1166.3873329162598 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.md0.in‚0( proc.memory.availableMB‚ 738.25390625 proc.cpu.threads‚17& disk.nvme0n1.out‚30852.76318359375' disk.nvme1c1n1.out‚20222.0009765625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚20222.0009765625 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0$ memory_percent‚95.49405574798584" proc.memory.rssMB‚ 3013.3984375Hñuž†‚(:“ ¤öÒ“äŠÌ gpu.0.gpu‚89.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚270.713' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.23766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d€Ä,2‡- _wandbruntime‚5861  _runtime‚5861Ú¦”ÜLjÁ trainer/global_step‚6499 % _timestamp‚1.7845126880296464e+09 _runtime‚5861.605232698 _step‚532 & lr-AdamW/pg1‚0.0004790038693908304 & lr-AdamW/pg2‚0.0004790038693908304Ï„MÌ‰Æ & lr-AdamW/pg2‚0.0004790038693908304 trainer/global_step‚6499 % _timestamp‚1.7845126880296464e+09 _runtime‚5861.605232698 _step‚532 & lr-AdamW/pg1‚0.0004790038693908304”ætˆR2Š- _wandbruntime‚5861  _runtime‚5861Å¿Fð'‹êO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26897152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚994897358032384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚994897340993024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64316081 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1002447507741184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚996613757131281 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1002447149277696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26897438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64305938 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚533 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚223754281896960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7593037806080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1002447149277696 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚5861.606133299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7550149708800 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7549808284672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15554683 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116939414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223754407875584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11342469 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15554966 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚216161244090880Y@ü•õ'ŒïO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚994897340993024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚994897358032384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚994897340993024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64316081 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1002447507741184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚996613757131281 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1002447149277696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26897438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64305938 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚533 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚223754281896960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7593037806080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1002447149277696 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚5861.606133299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7550149708800 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7549808284672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15554683 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116939414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223754407875584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11342469 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15554966 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚216161244090880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26897152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚996762296187117 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64325751 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15557290 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64315594 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64315594 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26901496 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚ 5862.21448714 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216194424360960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26901208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7550918383616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7551259814912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7550918383616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15557005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚989222428501632 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚989222411462272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116956877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845126886386704e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52631124 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116946718 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚995045615495168/›¤sÿ&ùM T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1002596533878784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64325751 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52631124 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64325751 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15557290 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64315594 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64315594 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26901496 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚ 5862.21448714 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚216194424360960 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26901208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7550918383616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7551259814912 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7550918383616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15557005 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚989222428501632 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚989222411462272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116956877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845126886386704e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52631124 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116946718 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚995045615495168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7539527124099 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116946718 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚996762296187117 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53012983 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64790590 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64790590 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚996446514168400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27099349 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚5905.442683981 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7594242510529 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64800733 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11429169 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7605377386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1009916443486720 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1002311083139584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1009916801950208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚6549 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1002311066100224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1002311083139584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚225469270934016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11429172 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7593901949255 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27099057 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7605718810624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚996446531207760 % _timestamp‚1.7845127318668685e+09 _step‚537 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15669888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53012981 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53012983 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64790590 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64790590 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚996446514168400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27099349 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚5905.442683981 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7594242510529 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64800733 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11429169 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7605377386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1009916443486720 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1002311083139584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1009916801950208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚6549 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1002311066100224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1002311083139584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22546¡÷ë­‰9270934016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11429172 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7593901949255 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27099057 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7605718810624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚996446531207760 % _timestamp‚1.7845127318668685e+09 _step‚537 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15669888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53012981 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117831179 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53020774 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚297 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚5906.075686061 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚538 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53020776 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64800246 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1002459340602368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7606487485440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11430906 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15672181 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7606828916736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27103084 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1002459357641728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15672478 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117831179 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53020774 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27103384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64810403 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64800246 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217853456477184 trainer/global_step‚6549 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7650005781504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117821020 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64810403 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11430903 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1010065828087808 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1010066186558464 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1002459340602368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1010066186558464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53020776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1004188955173379 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚225503443646464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7595010337507œ03@'¡€N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚225503567520768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217853437864960 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7650111043584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117821020 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845127324998167e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7606487485440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7595350898893 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53020774 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚297 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚5906.075686061 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚538 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53020776 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64800246 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1002459340602368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7606487485440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11430906 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15672181 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7606828916736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27103084 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1002459357641728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15672478 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117831179 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53020774 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27103384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64810403 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64800246 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚217853456477184 trainer/global_step‚6549 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7650005781504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117821020 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64810403 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11430903 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1010065828087808 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1010066186558464 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1002459340602368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1010066186558464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53020776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1004188955173379 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚225503443646464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7595010337507 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚996593944835872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117831179 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1017385737695744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_ó”eo!batch_start/inactive_split.small_pool.current‚291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 _runtime‚ 5949.4980095 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53402633 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65275242 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7660946488320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7704725326848 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1017385737695744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚541 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53402633 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1009724808246784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65275242 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1003818064581360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15785229 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11515872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845127759225242e+09 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7660946488320 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27301392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27301098 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1017386096159232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118688018>µèÇü'°öO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1017386096159232 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1009724791207424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1011467790305297 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53402631 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚219479271098880 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6599 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1017385737695744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 _runtime‚ 5949.4980095 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53402633 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65275242 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7660946488320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7704725326848 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1017385737695744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚541 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53402633 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1009724808246784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65275242 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1003818064581360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15785229 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11515872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845127759225242e+09 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7660946488320 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27301392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27301098 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1017386096159232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118688018MØþŸ2±- _wandbruntime‚5950  _runtime‚5950m…>'²ýM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7650834112301 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7650493550915 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1017535480767488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _runtime‚5950.120153392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11517606 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118695322 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118705481 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11517603 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚227218418833408 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53410424 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27305056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7705948852224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7705835201536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1011616329361133 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1003965495248832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118705481 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53410424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1009873082748928 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1009873065709568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚227218286570496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚219512451368960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1017535480767488 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1017535122296832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7662056587264 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53410426 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15787453 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚6599 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184‚¹0z=³‚N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118695322 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1011615971760387 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 % _timestamp‚1.7845127765443208e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65284898 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7662398018560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27305056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7705948852224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7705835201536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1011616329361133 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1003965495248832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118705481 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53410424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1009873082748928 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1009873065709568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚227218286570496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚219512451368960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1017535480767488 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1017535122296832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7662056587264 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53410426 L>DeviceStatsMonitor.on_train_bÅmŸËatch_end/active_bytes.all.current‚ 358470656 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15787453 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚6599 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7650834112301 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7650493550915 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1017535480767488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _runtime‚5950.120153392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11517606 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118695322 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118705481 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11517603 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚227218418833408 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53410424 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 B÷¾œ¸‚(:‘ ‚öÒçȱÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.155' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.05166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“MÖg¹‚(:Ü œ‚öÒ×í¿Ë" disk.dm-0.out‚45080.2978515625 disk.dm-1.in‚0" disk.dm-1.out‚9187.33056640625 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚96.99771404266357( proc.memory.availableMB‚ 491.89453125 cpu‚25.659653992211837 network.recv‚ 1.593979e+06( disk.nvme0c0n1.out‚34013.05810546875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚20642.4580078125 disk.md0.out‚ 181.51171875 proc.cpu.threads‚17 network.sent‚ 1.0507683e+07) disk./.usagePercent‚76.00467484969111 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836% disk./.usageGB‚1166.8617706298828 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚34013.05810546875% disk.nvme1n1.out‚20642.4580078125 disk.md0.in‚0ª©Ìžº‚(:“ œ‚öÒ¢á™Ì gpu.0.gpu‚87.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚277.634' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.54466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘|CxZ»‚(:Ï «‚öÒŸ«ÔË) disk./.usagePercent‚76.00467758290525 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625" disk.dm-0.out‚45080.8408203125 network.recv‚ 1.597526e+06( disk.nvme0c0n1.out‚34013.76806640625 disk.nvme1n1.in‚0" disk.dm-1.out‚9187.37744140625( proc.memory.availableMB‚ 491.89453125" proc.memory.rssMB‚ 3013.3984375 cpu‚25.660503223565982 network.sent‚ 1.0514379e+07% disk./.usageGB‚1166.8618125915527& disk.nvme0n1.out‚34013.76806640625! disk.nvme1c1n1.out‚ 20642.8125 disk.nvme1n1.out‚ 20642.8125 disk.md0.in‚0 disk.md0.out‚ 181.7265625* proc.memory.percent‚18.392324447631836 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0$ memory_percent‚96.99771404266357 proc.cpu.threads‚17ìoÝcœ¼‚(:‘ «‚öÒÿÈ‘Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.96' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.986666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m]ôÃ2½- _wandbruntime‚5993  _runtime‚5993ÙG›Cɾà _runtime‚5993.925601335 _step‚544 ' lr-AdamW/pg1‚0.00047282988954569825 ' lr-AdamW/pg2‚0.00047282988954569825 trainer/global_step‚6649 % _timestamp‚1.7845128203500361e+096MÎ¿È ' lr-AdamW/pg1‚0.00047282988954569825 ' lr-AdamW/pg2‚0.00047282988954569825 trainer/global_step‚6649 % _timestamp‚1.7845128203500361e+09 _runtime‚5993.925601335 _step‚544 °eÀ2À- _wandbruntime‚5993  _runtime‚5993Ÿ_§›ö'ÁðO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1018894449291671 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221138284602880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1024855390368256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15900282 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 $ _timestamp‚1.784512820351074e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53792281 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1017138516314624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1011189580915600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53792283 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1018894806892305 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27502572 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27502854 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6649 _runtime‚ 5993.92663603 _step‚545 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65759894 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7716857014272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65770037 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228898931601920 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1011189597954960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119552175 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1024855390368256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7760628386816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7705208937345 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11602569 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1017138516314624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11602572 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1017138533353984 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1017138533353984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65759894 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1024855031904768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53792281 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7760529409024ÓÕ{Eû'ÂõO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27502572 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27502854 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6649 _runtime‚ 5993.92663603 _step‚545 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65759894 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7716857014272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65770037 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228898931601920 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1011189597954960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119552175 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1024855390368256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7760628386816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7705208937345 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11602569 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1017138516314624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11602572 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1017138533353984 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1017138533353984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65759894 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1024855031904768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53792281 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7760529409024 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1018894449291671 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚221138284602880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1024855390368256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15900282 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 $ _timestamp‚1.784512820351074e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53792281 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1017138516314624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1011189580915600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53792283 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1018894806892305 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¡ZôÜõ2Ã- _wandbruntime‚5994  _runtime‚5994-ƒ UÄýM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 CÐâ~®&HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1019042988347395 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7717967120384 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119579783 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7717625689088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7761647473664 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65769550 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53800074 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119569624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7706317325709 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1017286790816768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7717625689088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228933112346624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1017286807856128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1019043345948141 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 trainer/global_step‚6649 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119579783 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65779707 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65779707 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119569624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65769550 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5994.548116486 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27506582 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1017286807856128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221171483485184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7705976764323 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15902555 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53800076 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7717967120384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7761748541440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27506861 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11604303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228933232026624 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1011337028622432 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚546 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¨ÿ¢¿'Å‚N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1017286807856128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1019043345948141 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 trainer/global_step‚6649 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119579783 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65779707 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65779707 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119569624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65769550 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5994.548116486 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27506582 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1017286807856128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221171483485184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7705976764323 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15902555 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53800076 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7717967120384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7761748541440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27506861 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11604303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228933232026624 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1011337028622432 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚546 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1019042988347395 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7717967120384 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119579783 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7717625689088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7761647473664 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65769550 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53800074 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119569624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7706317325709 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1017286790816768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7717625689088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228933112346624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1¢Æmw*2Æ- _wandbruntime‚5994  _runtime‚5994T>&Ç€ 2loss/FBt2/assignment_loss‚0.011130108908048926 /loss/FRt1/detection_loss‚0.4946614473746762 /loss/FRt2/detection_loss‚1.4787781099608956 1loss/SRqqt1/detection_loss‚0.7309021191059858 .loss/FBt2/detection_loss‚0.494712213435295 2loss/SRqqt1/assignment_loss‚0.4178976757301698 2loss/FBt1/assignment_loss‚0.010688267324663285 %loss/total_loss‚7.832118957552165 epoch‚16 _step‚547 /loss/FRt1/assignment_loss‚1.634329473445738 1loss/SRqqt2/assignment_loss‚0.593628097462684 /loss/FBt1/detection_loss‚0.5005079918950797 _runtime‚5994.548636978 0loss/FRt2/assignment_loss‚0.8026829936102142 1loss/SRqqt2/detection_loss‚0.6622004592987146 trainer/global_step‚6649 $ _timestamp‚1.784512820973356e+09tèÌP È… 2loss/FBt1/assignment_loss‚0.010688267324663285 %loss/total_loss‚7.832118957552165 epoch‚16 _step‚547 /loss/FRt1/assignment_loss‚1.634329473445738 1loss/SRqqt2/assignment_loss‚0.593628097462684 /loss/FBt1/detection_loss‚0.5005079918950797 _runtime‚5994.548636978 0loss/FRt2/assignment_loss‚0.8026829936102142 1loss/SRqqt2/detection_loss‚0.6622004592987146 trainer/global_step‚6649 $ _timestamp‚1.784512820973356e+09 2loss/FBt2/assignment_loss‚0.011130108908048926 /loss/FRt1/detection_loss‚0.4946614473746762 /loss/FRt2/detection_loss‚1.4787781099608956 1loss/SRqqt1/detection_loss‚0.7309021191059858 .loss/FBt2/detection_loss‚0.494712213435295 2loss/SRqqt1/assignment_loss‚0.4178976757301698£`‹øÉ‚(:’ º‚öÒ¬‡¼Ë gpu.0.gpu‚10.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.218' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.07266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u¤ Ü\Ê‚(:Ñ º‚öÒíÌåË) disk./.usagePercent‚76.00467559511314! disk.dm-1.out‚9187.6064453125' disk.nvme1c1n1.out‚20645.6669921875 disk.nvme1n1.in‚0 disk.md0.out‚ 182.39453125! disk.dm-0.out‚45092.154296875' proc.memory.availableMB‚ 491.6171875 proc.cpu.threads‚17 network.sent‚ 1.0530116e+07 network.recv‚ 1.60023e+06 disk.rbd0.out‚0% disk.nvme0c0n1.out‚34023.87109375 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625# memory_percent‚96.9994068145752* proc.memory.percent‚18.392324447631836 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625# disk.nvme0n1.out‚34023.87109375 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚20645.6669921875 cpu‚25.660974483176165% disk./.usageGB‚1166.8617820739746 disk.nvme0n1.in‚ 0.00390625" proc.memory.rssMB‚ 3013.3984375'(̃gË‚(:Ü É‚öÒà•¨Ë disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚20646.19580078125 disk.md0.in‚0$ memory_percent‚96.99790477752686 proc.cpu.threads‚17 network.sent‚ 1.0580838e+07! disk.dm-1.out‚9187.6533203125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚34024.74365234375& disk.nvme1n1.out‚20646.19580078125 disk.dm-0.in‚ 0.00390625 cpu‚25.660454071544777) disk./.usagePercent‚76.00468205543746 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚34024.74365234375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 182.75390625( proc.memory.availableMB‚ 491.86328125 network.recv‚ 1.605779e+06% disk./.usageGB‚1166.8618812561035! disk.dm-0.out‚45092.732421875" proc.memory.rssMB‚ 3013.3984375* proc.memory.percent‚18.392324447631836óA3“Ì‚(:ˆ É‚öÒäÄöË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.488' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.496 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ•›_Í‚(:Ô Ø‚öÒýÄžË disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚34027.38232421875 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0& disk.nvme1n1.out‚20646.85498046875 disk.md0.in‚0 disk.dm-0.out‚ 45095.5625) disk./.usagePercent‚76.00465273550407 disk.rbd0.in‚0 disk.md0.out‚ 182.93359375 disk.dm-1.in‚0$ memory_percent‚96.99790477752686( proc.memory.availableMB‚ 491.86328125" proc.memory.rssMB‚ 3013.3984375 cpu‚25.659808570102197 network.sent‚ 1.058376e+07 disk.nvme0n1.in‚ 0.00390625 disk.dm-0.in‚ 0.00390625* proc.memory.percent‚18.392324447631836 proc.cpu.threads‚17 network.recv‚ 1.609428e+06% disk./.usageGB‚1166.8614311218262& disk.nvme0n1.out‚34027.38232421875( disk.nvme1c1n1.out‚20646.85498046875 disk.dm-1.out‚ 9187.70703125Šñ©7žÎ‚(:“ Ø‚öÒÛýòË gpu.0.gpu‚94.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚274.198' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.39933333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v7;Þ2Ï- _wandbruntime‚6036  _runtime‚6036ÑFS>ÇÐÁ & lr-AdamW/pg2‚0.0004707504488739806 trainer/global_step‚6699 % _timestamp‚1.7845128627727191e+09 _runtime‚6036.348288157 _step‚548 & lr-AdamW/pg1‚0.0004707504488739806ù¨‚òÌÑÆ & lr-AdamW/pg1‚0.0004707504488739806 & lr-AdamW/pg2‚0.0004707504488739806 trainer/global_step‚6699 % _timestamp‚1.7845128627727191e+09 _runtime‚6036.348288157 _step‚548¤‰È82Ò- _wandbruntime‚6036  _runtime‚6036˜›ˆb°ÓóO ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1024552258461184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66254689 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1032324326113792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 _step‚549 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚222797298106880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16015023 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1026321823479313 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54181933 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1024552241421824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27704015 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16014746 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66254689 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230613651210752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54181931 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7772084691968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27704295 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7772426116096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120436622 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1018561114289200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7760692150753 % _timestamp‚1.7845128627736163e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1032324326113792 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1024552258461184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230613775092224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11689269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11689272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7760351589479“¶ ‰þ'ÔøO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1032324326113792 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1024552258461184 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230613775092224 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11689269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11689272 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7760351589479 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1024552258461184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66254689 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1032324326113792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 _step‚549 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚222797298106880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16015023 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1026321823479313 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54181933 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1024552241421824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27704015 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16014746 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66254689 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230613651210752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54181931 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7772084691968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27704295 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7772426116096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120436622 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1018561114289200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7760692150753 % _timestamp‚1.7845128627736163e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0¥rÞ¢"2Õ- _wandbruntime‚6036  _runtime‚6036šdà5'ÖüM N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120454085 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16017270 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66264359 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7817571093504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1024700515923968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222830478376960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54189726 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27707998 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1026470362535149 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 % _timestamp‚1.7845128633834395e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27708276 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚550 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120443926 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1032474069185536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7773536222208 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7773536222208 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7773194790912 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1024700532963328 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120443926 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222830496989184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452ST«b#×N UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120454085 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16017270 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66264359 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7817571093504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1024700515923968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222830478376960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54189726 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27707998 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1026470362535149 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 % _timestamp‚1.7845128633834395e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27708276 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚550 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120443926 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1032474069185536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7773536222208 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7773536222208 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7773194790912 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1024700532963328 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120443926 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222830496989184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1039793620322816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66739341 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232328529354752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54571583 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1039793978786304 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11775969 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚6078.833624512 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7827995217920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66739341 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1031965983568384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66729198 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1031965966529024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16129269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1025932664702160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1033748482465687 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1031965966529024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54571581 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1033748840066321 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _step‚553 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7816175364161 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 121300779 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1039793978786304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224456330223104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7872318818816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7815834802887 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224456311610880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66729198 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1039793620322816 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7827653793792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54571583 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7827653793792 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54571581 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121310924 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1025932647662800 trainer/global_step‚6749®Nþ'æøO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121310924 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1025932647662800 trainer/global_step‚6749 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7827995217920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1039793620322816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66739341 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232328529354752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54571583 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1039793978786304 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11775969 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚6078.833624512 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7827995217920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66739341 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1031965983568384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66729198 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1031965966529024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16129269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1025932664702160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1033748482465687 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1031965966529024 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54571581 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1033748840066321 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _step‚553 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7816175364161 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 121300779 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1039793978786304 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224456330223104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7872318818816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7815834802887 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224456311610880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66729198 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1039793620322816 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7827653793792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54571583 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7827653793792 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54571581 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0©ÜHg2ç- _wandbruntime‚6079  _runtime‚6079¤#Ɖ‚èýM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66738854 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224489491880960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27909189 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7817283752525 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1032114258070528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1033897379122157 % _timestamp‚1.7845129058727355e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121328387 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1039943363394560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66749011 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7829105324032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7873331706368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7829105324032 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1033897021521411 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_bñCúÇatch_end/active_bytes.all.current‚ 358470656 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66738854 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16131771 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232362947461632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6079.448473028 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224489510493184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1032114241031168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121328387 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7873436968448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1026080078330272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121318228 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1026080095369632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1039943004923904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121318228 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11777703 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7828763892736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16131486 trainer/global_step‚6749 _step‚554 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54579374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54579374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7816943191139 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66738854 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224489491880960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27909189 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7817283752525 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1032114258070528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1033897379122157 % _timestamp‚1.7845129058727355e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121328387 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1039943363394560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66749011 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7829105324032 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7873331706368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7829105324032 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1033897021521411 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66738854 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16131771 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232362947461632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6079.448473028 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224489510493184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1032114241031168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121328387 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7873436968448 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1026080078330272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121318228 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1026080095369632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1039943004923904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121318228 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11777703 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7828763892736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16131486 trainer/global_step‚6749 _step‚554 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54579374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54579374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7816943191139 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 Em2ö- _wandbruntime‚6121  _runtime‚6121á«Où'÷óO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7928154158080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1047263272995328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1047262914531840 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67213850 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16243668 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1039379691636224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67223993 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1041175856653329 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54961231 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1033304198075760 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1039379708675584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7928042597376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1041175499052695 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28106337 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122175081 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7883564319744 _step‚557 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚234043367712256 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67223993 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11862672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1047262914531840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226115343727104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1033304181036400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7871318016295 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16243957 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67213850 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 % _timestamp‚1.7845129479368148e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 238736143361{nºÈøøO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54961231 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1033304198075760 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1039379708675584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7928042597376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1041175499052695 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28106337 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122175081 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TDDeviceStatsMonitor.on_train_batch_stK~ÁK6!art/active_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7883564319744 _step‚557 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚234043367712256 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67223993 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11862672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1047262914531840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226115343727104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1033304181036400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7871318016295 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16243957 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67213850 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 % _timestamp‚1.7845129479368148e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7928154158080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1047263272995328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1047262914531840 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67213850 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16243668 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1039379691636224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67223993 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1041175856653329 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184­0(à2ù- _wandbruntime‚6122  _runtime‚6122R$åñ'úýM FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122202689 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16245910 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28110313 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1047412299132928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122192530 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54969026 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67223506 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7929159248896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122192530 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1039527983177728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1047412657603584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54969024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54969026 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54969024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7884332994560 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1041324395709165 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28110594 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11864406 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1033451611703872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7929268705280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _step‚558 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122192530 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1039527983177728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1047412657603584 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54969024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54969026 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54969024 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7884332994560 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1041324395709165 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28110594 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11864406 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1033451611703872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7929268705280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _step‚558 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122202689 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16245910 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28110313 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1047412299132928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122192530 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54969026 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67223506 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7929159248896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799®¬±¸?2ü- _wandbruntime‚6122  _runtime‚61223•_0 ý… trainer/global_step‚6799 _runtime‚6122.126787847 2loss/SRqqt1/assignment_loss‚0.4500545288583541 /loss/FBt1/detection_loss‚0.4846619110349126 epoch‚16 %loss/total_loss‚7.481753419955041 0loss/FRt1/assignment_loss‚1.6004391410916121 0loss/SRqqt1/detection_loss‚0.674486041005583 % _timestamp‚1.7845129485513968e+09 _step‚559 2loss/SRqqt2/assignment_loss‚0.5168722332836747 0loss/SRqqt2/detection_loss‚0.686416696133925 0loss/FRt2/assignment_loss‚0.7014722831437153 3loss/FBt1/assignment_loss‚0.0066780577796773126 3loss/FBt2/assignment_loss‚0.0012586828170810804 /loss/FRt1/detection_loss‚0.5239744546516195 /loss/FRt2/detection_loss‚1.5473704516900282 0loss/FBt2/detection_loss‚0.28806893846485765ÁGaþŠ %loss/total_loss‚7.481753419955041 0loss/FRt1/assignment_loss‚1.6004391410916121 0loss/SRqqt1/detection_loss‚0.674486041005583 % _timestamp‚1.7845129485513968e+09 _step‚559 2loss/SRqqt2/assignment_loss‚0.5168722332836747 0loss/SRqqt2/detection_loss‚0.686416696133925 0loss/FRt2/assignment_loss‚0.7014722831437153 3loss/FBt1/assignment_loss‚0.0066780577796773126 3loss/FBt2/assignment_loss‚0.0012586828170810804 /loss/FRt1/detection_loss‚0.5239744546516195 /loss/FRt2/detection_loss‚1.5473704516900282 0loss/FBt2/detection_loss‚0.28806893846485765 trainer/global_step‚6799 _runtime‚6122.126787847 2loss/SRqqt1/assignment_loss‚0.4500545288583541 /loss/FBt1/detection_loss‚0.4846619110349126 epoch‚16¯° @cÿ‚(:Ø ÁƒöÒÉôÍË) disk./.usagePercent‚76.00754969400724# disk.dm-0.out‚47026.16162109375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625$ memory_percent‚96.99113368988037! proc.memory.rssMB‚ 3013.453125 network.sent‚ 1.0733779e+07 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚20653.38623046875 disk.md0.out‚ 185.7265625 disk.dm-1.out‚9188.498046875 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 1.637403e+06$ disk./.usageGB‚1166.905906677246 disk.nvme1n1.in‚0& disk.nvme1n1.out‚20653.38623046875 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625' disk.nvme0c0n1.out‚35958.2451171875% disk.nvme0n1.out‚35958.2451171875( proc.memory.availableMB‚ 492.97265625* proc.memory.percent‚18.392658233642578 cpu‚25.660775911104757 ýF–’€‚(:‡ ÁƒöÒ†¥ÔÌ gpu.0.gpu‚25.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.667' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.889 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%,ãg2- _wandbruntime‚6137  _runtime‚6137³D€Ì‚Åz 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6844800114631653 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48365938663482666 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6614583730697632 jet/accuracy_2_of_5‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6654528975486755 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8211262226104736 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8211262226104736 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6535664796829224 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2667691707611084 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7617896199226379 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.777254581451416 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9969269633293152 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7559233903884888 & Purity/2FRt1SRqqt*àS8FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8321239948272705 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7879124879837036 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 )loss/val_total_loss‚7.743695382928074 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6831007599830627 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 +jet/accuracy_3_of_4‚0.09237384796142578 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6875685453414917 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7215291261672974 ,jet/accuracy_4_of_4‚0.011098379269242287 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9958277344703674 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19049251079559326 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1772698163986206 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.236591637134552 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7572534680366516 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04563511162996292 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21356526017189026 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20053933560848236 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5759031176567078 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7121720314025879 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6353272199630737 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992209672927856 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912253022193909 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 /particle/accuracy_4_of_4‚0.9358518719673157 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6927905082702637 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930034875869751 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8047429919242859 6validation_average_jet_accuracy‚0.3793638050556183 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3230169713497162 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7180950045585632 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.40993571281433105 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.032370369881391525 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24115464091300964 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.032583799213171005 +particle/specificity‚0.9197947382926941 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7679440379142761 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15661349892616272 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14024397730827332 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6722844243049622 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9963346123695374 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7620531916618347 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2152256965637207 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.201316699385643 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 trainer/global_step‚6799 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7761065363883972 % _timestamp‚1.7845129643914125e+09 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.690554141998291 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9903279542922974 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5190625190734863 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.856185257434845 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21377122402191162 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20053933560848236 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7282559871673584 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8136875629425049 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9358115196228027 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2849455177783966 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7157585620880127 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 epoch‚16 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7256957292556763 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9875958561897278 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2230129987001419 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20474933087825775 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16611400246620178 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7644175291061401 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.856185257434845 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884729981422424 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48972466588020325 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3365776538848877 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.658300518989563 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6875685453414917 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7197927832603455 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6403892040252686 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0384000726044178 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19108156859874725 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17731092870235443 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_2‚0.8105250597000122 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_3_of_4‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5253757238388062 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.78125 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.036699067801237106 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03705117106437683 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 particle/accuracy_1_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 *jet/accuracy_1_of_4‚0.9725117683410645 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8214285969734192 jet/accuracy_1_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988561272621155 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6640617251396179 particle/accuracy_2_of_3‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02374403551220894 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02374403551220894 )jet/accuracy_2_of_3‚0.599610447883606 jet/accuracy_4_of_5‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9326813220977783 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6319430470466614 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8131304383277893 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965262413024902 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5411496162414551 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.023782189935445786 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7483617067337036 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_3_of_3‚0.0752871111035347 particle/accuracy_3_of_3‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.060382552444934845 *jet/accuracy_1_of_1‚0.3322301506996155 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8718287944793701 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1938561499118805 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15406696498394012 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14493730664253235 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1290818303823471 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23102562129497528 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.210030198097229 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7292129397392273 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.747146725654602 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19587485492229462 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17770834267139435 *jet/accuracy_2_of_4‚0.8061320185661316 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9229111671447754 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7825194001197815 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 _runtime‚6137.967884525 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8091080784797668 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9065418243408203 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1458333432674408 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 'particle/f_score‚0.6623378396034241 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.031486302614212036 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5827736854553223 (particle/accuracy‚0.8597857356071472 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6947828531265259 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +jet/accuracy_2_of_2‚0.16911788284778595 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.884599506855011 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40593788027763367 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7683190703392029 7"Purity/0FRt0SRqqt2FBt/event_purity‚0.98065584897995 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972710609436035 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.5023477673530579 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.559166669845581 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.994263231754303 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05109942704439163 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_4_of_5‚1 _step‚560 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6579894423484802 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.38660249114036566Uä—P=ƒÊz @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 _runtime‚6137.967884525 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8091080784797668 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9065418243408203 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1458333432674408 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 'particle/f_score‚0.6623378396034241 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.031486302614212036 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5827736854553223 (particle/accuracy‚0.8597857356071472 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6947828531265259 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 +jet/accuracy_2_of_2‚0.16911788284778595 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.884599506855011 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40593788027763367 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7683190703392029 7"Purity/0FRt0SRqqt2FBt/event_purity‚0.98065584897995 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972710609436035 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.5023477673530579 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.559166669845581 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.994263231754303 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05109942704439163 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_4_of_5‚1 _step‚560 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6579894423484802 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6844800114631653 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48365938663482666 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6614583730697632 jet/accuracy_2_of_5‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6654528975486755 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8211262226104736 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8211262226104736 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6535664796829224 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2667691707611084 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7617896199226379 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.777254581451416 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9969269633293152 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7559233903884888 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8321239948272705 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7879124879837036 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 )loss/val_total_loss‚7.743695382928074 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6831007599830627 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 +jet/accuracy_3_of_4‚0.09237384796142578 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6875685453414917 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7215291261672974 ,jet/accuracy_4_of_4‚0.011098379269242287 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9958277344703674 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19049251079559326 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1772698163986206 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.236591637134552 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7572534680366516 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04563511162996292 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21356526017189026 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20053933560848236 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5759031176567078 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7121720314025879 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6353272199630737 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992209672927856 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912253022193909 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 /particle/accuracy_4_of_4‚0.9358518719673157 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.5625 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6927905082702637 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930034875869751 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8047429919242859 6validation_average_jet_accuracy‚0.3793638050556183 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3230169713497162 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7180950045585632 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.40993571281433105 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.032370369881391525 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24115464091300964 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.032583799213171005 +particle/specificity‚0.9197947382926941 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7679440379142761 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15661349892616272 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14024397730827332 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6722844243049622 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9963346123695374 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7620531916618347 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2152256965637207 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.201316699385643 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 trainer/global_step‚6799 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7761065363883972 % _timestamp‚1.7845129643914125e+09 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.690554141998291 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9903279542922974 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5190625190734863 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.856185257434845 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21377122402191162 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20053933560848236 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7282559871673584 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8136875629425049 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9358115196228027 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2849455177783966 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7157585620880127 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 epoch‚16 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7256957292556763 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9875958561897278 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2230129987001419 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20474933087825775 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16611400246620178 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7644175291061401 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.856185257434845 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884729981422424 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48972466588020325 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3365776538848877 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.658300518989563 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6875685453414917 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7197927832603455 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.6428571343421936 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6403892040252686 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt1SRqqt0FBt/event_purity‚0.0384000726044178 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19108156859874725 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17731092870235443 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_2‚0.8105250597000122 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_3_of_4‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5253757238388062 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.78125 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.036699067801237106 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03705117106437683 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 particle/accuracy_1_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 *jet/accuracy_1_of_4‚0.9725117683410645 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8214285969734192 jet/accuracy_1_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988561272621155 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6640617251396179 particle/accuracy_2_of_3‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02374403551220894 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02374403551220894 )jet/accuracy_2_of_3‚0.599610447883606 jet/accuracy_4_of_5‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9326813220977783 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6319430470466614 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8131304383277893 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965262413024902 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5411496162414551 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.023782189935445786 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7483617067337036 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_3_of_3‚0.0752871111035347 particle/accuracy_3_of_3‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.060382552444934845 *jet/accuracy_1_of_1‚0.3322301506996155 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8718287944793701 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1938561499118805 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15406696498394012 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14493730664253235 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1290818303823471 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23102562129497528 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.210030198097229 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7292129397392273 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.747146725654602 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19587485492229462 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17770834267139435 *jet/accuracy_2_of_4‚0.8061320185661316 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9229111671447754 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7825194001197815 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875°o®ÞY„‚(:Î ЃöÒŒ²¤Ë disk.nvme1n1.in‚0' proc.memory.availableMB‚ 738.4140625 cpu‚25.6609146900417 proc.cpu.threads‚17 network.recv‚ 1.642165e+06$ disk./.usageGB‚1166.859302520752 disk.md0.in‚0 disk.md0.out‚ 186.04296875 disk.rbd0.out‚0 proc.memory.rssMB‚3013.5( proc.memory.percent‚18.3929443359375 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.dm-1.out‚9433.998046875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚36608.5927734375& disk.nvme1n1.out‚21119.92919921875$ memory_percent‚95.49307823181152 network.sent‚ 1.0796129e+07( disk./.usagePercent‚76.0045140870055# disk.dm-0.out‚47896.84912109375 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚36608.5927734375 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚21119.92919921875Èñ†7ž…‚(:“ ЃöÒ¿à“Ì gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.993' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.66433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00 5x\†‚(:Ñ ߃öÒàÀË disk.md0.in‚0# disk.dm-0.out‚47897.38037109375 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚36609.3662109375 disk.nvme1n1.in‚0( proc.memory.percent‚18.3929443359375 disk.md0.out‚ 186.3359375' disk.nvme0c0n1.out‚36609.3662109375 disk.nvme1c1n1.in‚0" memory_percent‚95.493483543396( proc.memory.availableMB‚ 738.34765625 network.sent‚ 1.0853751e+07) disk./.usagePercent‚76.00455384284739 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0! disk.dm-1.out‚10214.248046875 proc.memory.rssMB‚3013.5 cpu‚25.661412068218485% disk./.usageGB‚1166.8599128723145 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚21900.58544921875& disk.nvme1n1.out‚21900.58544921875 proc.cpu.threads‚17 disk.rbd0.out‚0 network.recv‚ 1.647236e+06öÔä;‡‚(:’ ߃öÒ’Ú‚Ì gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.61' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.53666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01[qˆ‚(:Ð îƒìºÒ KöÒ¯Ë¼Ë disk.md0.out‚ 186.890625 disk.dm-0.in‚ 0.00390625# disk.dm-1.out‚10214.40869140625 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚36617.2314453125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚36617.2314453125# disk.dm-0.out‚47906.63427734375 disk.dm-1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 738.33984375( proc.memory.percent‚18.3929443359375 network.recv‚ 1.650168e+06) disk./.usagePercent‚76.00455558216547% disk./.usageGB‚1166.8599395751953& disk.nvme1c1n1.out‚21903.310546875$ disk.nvme1n1.out‚21903.310546875 disk.md0.in‚0$ memory_percent‚95.49353122711182 cpu‚25.662272357070847 network.sent‚ 1.0856673e+07 disk.nvme1n1.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚3013.5 disk.nvme1c1n1.in‚0‹:‚¡ž‰‚(:“ îƒöÒÛê¢Ì gpu.0.gpu‚66.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚284.821' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.94033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ej2Š- _wandbruntime‚6182  _runtime‚6182òÞͨÈ‹Â _runtime‚6182.296973736 _step‚561 ' lr-AdamW/pg1‚0.00046444965444899936 ' lr-AdamW/pg2‚0.00046444965444899936 trainer/global_step‚6849 $ _timestamp‚1.784513008720944e+09VœÊ¡ÍŒÇ ' lr-AdamW/pg1‚0.00046444965444899936 ' lr-AdamW/pg2‚0.00046444965444899936 trainer/global_step‚6849 $ _timestamp‚1.784513008720944e+09 _runtime‚6182.296973736 _step‚561±zp2- _wandbruntime‚6182  _runtime‚6182[ ?£ù'ŽóO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123253690 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚6849 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1056694870454784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28345716 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16392523 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235801598360064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8004555100160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67780001 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55473689 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7959451259904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123263835 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1056695228918272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67790144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚235801472381440 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1050544433097463 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1042596970418224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7947105078605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1048735777658368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1056694870454784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227796917281280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67780001 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16392231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67790144 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1048735760619008 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1042596987457584 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚11805åþ'øO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1056695228918272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55473689 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7959451259904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8004662466560 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1056694870454784 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227796917281280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67780001 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16392231 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67790144 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1048735760619008 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1042596987457584 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123253690 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚6849 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1056694870454784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28345716 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16392523 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235801598360064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8004555100160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67780001 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55473689 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7959451259904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123263835 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1056695228918272 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67790144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚235801472381440 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1050544433097463 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1042596970418224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7947105078605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1048735777658368 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 B'‘üM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67799814 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55481484 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11955222 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1048884052160512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227830116163584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7960561366016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7960219934720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1048884035121152 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235835781677056 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55481482 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚6182.918198899 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1056844613526528 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28349697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16394478 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1050692614552553 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28349987 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚227830097551360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 % _timestamp‚1.7845130093423975e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123281298 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67789657 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚563 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7948554028243 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67799814 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1048884052160512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123271139 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8005774707712 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1042744418125056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7948213466857 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123281298 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123271139 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1048884035121152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0‡°,¦’N JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1048884035121152 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 áꀷa%D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235835781677056 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55481482 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚6182.918198899 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1056844613526528 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28349697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16394478 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1050692614552553 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28349987 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚227830097551360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 % _timestamp‚1.7845130093423975e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123281298 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67789657 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚563 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7948554028243 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67799814 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1048884052160512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123271139 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8005774707712 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1042744418125056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7948213466857 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123281298 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123271139 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1048884035121152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67799814 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55481484 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11955222 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1048884052160512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227830116163584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7960561366016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7960219934720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452³ñÞeÜ2“- _wandbruntime‚6182  _runtime‚61820• ”† 3loss/SRqqt2/assignment_loss‚0.49097754009549827 epoch‚17 _runtime‚6182.919361081 4loss/FBt2/assignment_loss‚2.7124338948290133e-05 /loss/FRt2/detection_loss‚1.5193337290104842 /loss/FBt2/detection_loss‚0.2998978745613602 &loss/total_loss‚7.4456026275830265 0loss/FRt1/assignment_loss‚1.6890769590264403 2loss/SRqqt1/assignment_loss‚0.4490689097130861 2loss/FBt1/assignment_loss‚0.013130602147577672 /loss/FBt1/detection_loss‚0.4032331529608744 trainer/global_step‚6849 $ _timestamp‚1.784513009343951e+09 _step‚564 0loss/FRt2/assignment_loss‚0.7188666545763248 .loss/FRt1/detection_loss‚0.489889306825921 1loss/SRqqt1/detection_loss‚0.7561154836332379 1loss/SRqqt2/detection_loss‚0.6159852906932729òñ•‹ 4loss/FBt2/assignment_loss‚2.7124338948290133e-05 /loss/FRt2/detection_loss‚1.5193337290104842 /loss/FBt2/detection_loss‚0.2998978745613602 &loss/total_loss‚7.4456026275830265 0loss/FRt1/assignment_loss‚1.6890769590264403 2loss/SRqqt1/assignment_loss‚0.4490689097130861 2loss/FBt1/assignment_loss‚0.013130602147577672 /loss/FBt1/detection_loss‚0.4032331529608744 trainer/global_step‚6849 $ _timestamp‚1.784513009343951e+09 _step‚564 0loss/FRt2/assignment_loss‚0.7188666545763248 .loss/FRt1/detection_loss‚0.489889306825921 1loss/SRqqt1/detection_loss‚0.7561154836332379 1loss/SRqqt2/detection_loss‚0.6159852906932729 3loss/SRqqt2/assignment_loss‚0.49097754009549827 epoch‚17 _runtime‚6182.919361081´IÑVfQ–‚(:Æ ýƒöÒþοË$ memory_percent‚95.49407958984375) disk./.usagePercent‚76.00454415236092 disk.dm-1.in‚0 disk.md0.out‚ 187.32421875 disk.dm-0.in‚ 0.00390625! disk.dm-0.out‚47907.361328125( proc.memory.percent‚18.3929443359375 network.sent‚ 1.0875711e+07# disk.dm-1.out‚10217.38134765625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625& disk.nvme1n1.out‚21906.84033203125" proc.memory.availableMB‚738.25 network.recv‚ 1.65406e+06$ disk./.usageGB‚1166.859764099121$ disk.nvme0c0n1.out‚ 36618.3359375 disk.nvme1n1.in‚0 disk.md0.in‚0 proc.memory.rssMB‚3013.5 cpu‚25.663208498822037 proc.cpu.threads‚17 disk.rbd0.in‚0" disk.nvme0n1.out‚ 36618.3359375 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚21906.84033203125qÇ6ž—‚(:“ ýƒöÒÄÌ gpu.0.gpu‚78.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚270.012' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.00399999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ynHX˜‚(:Í Œ„öÒ„ÃÌË proc.cpu.threads‚17 network.recv‚ 1.659211e+06) disk./.usagePercent‚76.00458018109263% disk./.usageGB‚1166.8603172302246 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚21908.0830078125 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚36621.16162109375 disk.md0.out‚ 187.578125 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0" proc.memory.availableMB‚738.25 proc.memory.rssMB‚3013.5 cpu‚25.664059171257737 network.sent‚ 1.0926239e+07 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚21908.0830078125 disk.dm-1.in‚0# disk.dm-1.out‚10217.55126953125$ memory_percent‚95.49407958984375& disk.nvme0n1.out‚36621.16162109375" disk.dm-0.out‚47910.7001953125( proc.memory.percent‚18.3929443359375èºÌ(›™‚(: Œ„öÒÔ¾Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚259.37' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.45666666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sZÎ\š‚(:Ñ ›„öÒ¹ÙÃË network.sent‚ 1.0931075e+07( disk.nvme0c0n1.out‚36621.98291015625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.md0.in‚0" disk.dm-0.out‚47911.3291015625 disk.dm-1.in‚0 network.recv‚ 1.663036e+06 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚36621.98291015625& disk.nvme1c1n1.out‚21908.556640625# disk.dm-1.out‚10217.60595703125 proc.cpu.threads‚17 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚21908.556640625 disk.rbd0.out‚0( proc.memory.availableMB‚ 738.00390625 proc.memory.rssMB‚3013.5 cpu‚25.66494726928758) disk./.usagePercent‚76.00457645398245% disk./.usageGB‚1166.8602600097656 disk.md0.out‚ 187.8515625 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0$ memory_percent‚95.49558162689209( proc.memory.percent‚18.3929443359375Xk›‚(:’ ›„öÒЦ†Ì gpu.0.gpu‚14.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.441' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.14699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì®.2œ- _wandbruntime‚6226  _runtime‚6226*îŸìÆÀ & lr-AdamW/pg1‚0.0004623290151586389 & lr-AdamW/pg2‚0.0004623290151586389 trainer/global_step‚6899 % _timestamp‚1.7845130529570525e+09 _runtime‚ 6226.53266922 _step‚565o¦€ß˞Š_runtime‚ 6226.53266922 _step‚565 & lr-AdamW/pg1‚0.0004623290151586389 & lr-AdamW/pg2‚0.0004623290151586389 trainer/global_step‚6899 % _timestamp‚1.7845130529570525e+09µ¸Ùd2Ÿ- _wandbruntime‚6226  _runtime‚6226V"'ù' óO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6226.537078091 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1056149502765568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124127992 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68274796 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8060521796608 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8002588292013 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55863339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12040185 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8014678937600 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1057971449684471 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8002928853287 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 _step‚566 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1049968503791824 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1056149485726208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229455930785280 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55863339 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68264653 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1064164164663808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68274796 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1064164164663808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845130529581814e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8015020361728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8015020361728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16506614 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12040188 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16506897 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8060418624512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚6899 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28546799 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55863341 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143Ä¡@0¡øO VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1064164164663808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68274796 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1064164164663808 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845130529581814e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8015020361728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8015020361728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16506614 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12040188 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16506897 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8060418624512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚6899 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28546799 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55863341 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6226.537078091 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1056149502765568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.aèE¥ç ll.freed‚ 124127992 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68274796 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8060521796608 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8002588292013 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55863339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12040185 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8014678937600 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1057971449684471 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8002928853287 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 _step‚566 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1049968503791824 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1056149485726208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229455930785280 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55863339 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68264653¶,D‹z2¢- _wandbruntime‚6227  _runtime‚6227síƒ×'£ýM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28551022 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8015789036544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1058119988740307 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229489129667584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124155600 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28550731 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68284466 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845130535790682e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124155600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _runtime‚6227.154806527 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55871132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚237550771655680 _step‚567 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1056297777267712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1050115934459296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1064313907735552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124145441 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8061532531712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1064313549264896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68274309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16508812 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8061641988096 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1050115951498656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1056297777267712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8015789036544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68284466 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55871132 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8003696680265 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229489111055360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8004037241651 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12041922 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124145441 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647^0 Ó'¤‚N NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68284466 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845130535790682e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124155600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _runtime‚6227.154806527 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55871132 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚237550771655680 _step‚567 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1056297777267712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1050115934459296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1064313907735552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124145441 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8061532531712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1064313549264896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68274309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16508812 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8061641988096 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1050115951498656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1056297777267712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8015789036544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68284466 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55871132 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8003696680265 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229489111055360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8004037241651 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12041922 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124145441 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28551022 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8015789036544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1058119988740307 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚229489129667584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124155600 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28550731 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0·×…á`2¥- _wandbruntime‚6227  _runtime‚6227u Û ¦‡ 0loss/FRt1/assignment_loss‚1.6336350029771298 0loss/FRt2/assignment_loss‚0.7670085843839601 _runtime‚6227.155645839 2loss/FBt1/assignment_loss‚0.002822500770111781 3loss/FBt2/assignment_loss‚0.0009059855952446109 0loss/FRt1/detection_loss‚0.49454515151366574 1loss/SRqqt2/detection_loss‚0.6303418477228354 0loss/FBt2/detection_loss‚0.42614551607402973 %loss/total_loss‚7.549667832066684 _step‚568 2loss/SRqqt1/assignment_loss‚0.4592792092773902 2loss/SRqqt2/assignment_loss‚0.5039520147897578 /loss/FRt2/detection_loss‚1.5335872806034574 1loss/SRqqt1/detection_loss‚0.6934655718378594 /loss/FBt1/detection_loss‚0.4039791665212424 epoch‚17 % _timestamp‚1.7845130535801601e+09 trainer/global_step‚6899G¸“§Œ 0loss/FRt1/assignment_loss‚1.6336350029771298 0loss/FRt2/assignment_loss‚0.7670085843839601 _runtime‚6227.155645839 2loss/FBt1/assignment_loss‚0.002822500770111781 3loss/FBt2/assignment_loss‚0.0009059855952446109 0loss/FRt1/detection_loss‚0.49454515151366574 1loss/SRqqt2/detection_loss‚0.6303418477228354 0loss/FBt2/detection_loss‚0.42614551607402973 %loss/total_loss‚7.549667832066684 _step‚568 2loss/SRqqt1/assignment_loss‚0.4592792092773902 2loss/SRqqt2/assignment_loss‚0.5039520147897578 /loss/FRt2/detection_loss‚1.5335872806034574 1loss/SRqqt1/detection_loss‚0.6934655718378594 /loss/FBt1/detection_loss‚0.4039791665212424 epoch‚17 % _timestamp‚1.7845130535801601e+09 trainer/global_step‚6899¸üFhh\¨‚(:Ñ ª„öÒê°¶Ë disk.md0.in‚0# disk.dm-1.out‚10217.85302734375$ memory_percent‚96.99747562408447( proc.memory.availableMB‚ 491.93359375& disk.nvme0n1.out‚36623.21044921875 disk.nvme1n1.in‚0" disk.dm-0.out‚47912.1376953125 disk.dm-1.in‚0 proc.memory.rssMB‚3013.5( proc.memory.percent‚18.3929443359375 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 0.00390625 disk.dm-0.in‚ 0.00390625 cpu‚25.665791450755403 network.recv‚ 1.666485e+06) disk./.usagePercent‚76.00456403028187% disk./.usageGB‚1166.8600692749023 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚21909.48046875# disk.nvme1n1.out‚21909.48046875 disk.md0.out‚ 188.36328125 network.sent‚ 1.0933997e+07 disk.rbd0.in‚0( disk.nvme0c0n1.out‚36623.21044921875 disk.nvme0n1.in‚ 0.003906254©sZ“©‚(:ˆ ª„öÒÊÕ«Ì gpu.0.gpu‚41.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚273.945' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.315 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ì Ç]ª‚(:Ò ¹„öÒŒÒ£Ë network.sent‚ 1.0998283e+07( disk./.usagePercent‚76.0045689997621 disk.nvme0n1.in‚ 0.00390625 disk.md0.out‚ 188.85546875# disk.dm-0.out‚47913.34619140625# disk.dm-1.out‚10217.90380859375$ proc.memory.availableMB‚491.9375 proc.memory.rssMB‚3013.5& disk.nvme0n1.out‚36624.83154296875( disk.nvme1c1n1.out‚21910.17041015625 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625 cpu‚25.66679141927925 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 network.recv‚ 1.671535e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚36624.83154296875 disk.md0.in‚0$ memory_percent‚96.99745178222656% disk./.usageGB‚1166.8601455688477& disk.nvme1n1.out‚21910.17041015625 disk.dm-1.in‚0( proc.memory.percent‚18.3929443359375½÷¢ ž«‚(:“ ¹„öÒà•íË gpu.0.gpu‚88.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚278.474' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.82466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éx!^¬‚(:Ó È„öÒ®÷ÌË disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚47914.17041015625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme0n1.out‚36625.5732421875 network.recv‚ 1.674779e+06% disk./.usageGB‚1166.8607711791992 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚21910.8701171875$ memory_percent‚96.98989391326904( proc.memory.availableMB‚ 493.17578125) disk./.usagePercent‚76.00460974950003# disk.dm-1.out‚10217.99072265625 cpu‚25.66762353402752 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚ 189.09765625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚21910.8701171875 proc.memory.rssMB‚3013.5( proc.memory.percent‚18.3929443359375' disk.nvme0c0n1.out‚36625.5732421875 network.sent‚ 1.1001205e+07îkTNž­‚(:“ È„öÒÉ€£Ì gpu.0.gpu‚41.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚292.713' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.57100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íßë2®- _wandbruntime‚6270  _runtime‚6270¹¡ãÒǯÁ _runtime‚6270.414040697 _step‚569 ' lr-AdamW/pg1‚0.00046019841509321716 ' lr-AdamW/pg2‚0.00046019841509321716 trainer/global_step‚6949 # _timestamp‚1.78451309683495e+09ZyÍÌ°Æ _runtime‚6270.414040697 _step‚569 ' lr-AdamW/pg1‚0.00046019841509321716 ' lr-AdamW/pg2‚0.00046019841509321716 trainer/global_step‚6949 # _timestamp‚1.78451309683495e+09¹u¯Ù'2±- _wandbruntime‚6270  _runtime‚6270— …0 ²óO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚6270.414130757 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1063563210833408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1057340054204784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8058412066695 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚570 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56252989 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12126888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1065398108670845 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8116264545280 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125012439 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16621514 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16621220 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845130968356428e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1071633458872832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1063563227872768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28748105 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1057340037165424ñíÔÜþ'³øO ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1063563227872768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1063563210833408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8070589463552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239231208834560 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1071633458872832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1063563227872768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28748105 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1057340037165424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚6270.414130757 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1063563210833408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1057340054204784 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8058412066695 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚570 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56252989 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12126888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1065398108670845 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8116264545280 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125012439 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16621514 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16621220 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845130968356428e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚239265629892608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1063711485335552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8071699569664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8071358138368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239265514406912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231148124559360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚6949 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1057487467832896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8059179893673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1063711502374912 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8117486721024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68769118 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 % _timestamp‚1.7845130974608727e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12128622 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16623523 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8059520455059 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125029902 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125029902 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68758961 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1065547005327315 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28752434 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1071782843473920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68769118 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1057487484872256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28752142 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6271.036639466 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56260782 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534ä˜gâ¶N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125019743 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16623812 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1065546647726569 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1071783201944576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125019743 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56260782 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1057487467832896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8059179893673 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1063711502374912 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8117486721024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68769118 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 % _timestamp‚1.7845130974608727e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12128622 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16623523 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8059520455059 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125029902 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125029902 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.frC Ô4%eed‚68758961 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1065547005327315 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28752434 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1071782843473920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68769118 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1057487484872256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28752142 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6271.036639466 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56260782 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _step‚571 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56260784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8117389847552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚239265629892608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1063711485335552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8071699569664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8071358138368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239265514406912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231148124559360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚6949 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1079102753081856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56642639 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69244100 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125876596 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28949748 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12213585 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8113895280103 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _step‚574 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8125817141248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784513141400015e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8113554718829 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12213588 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚6314.975547343 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56642641 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8125817141248 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1079103111545344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8172213197824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1070976952979968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚6999 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1070976952979968 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1070976935940608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69233957 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69233957 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1070976935940608 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1072825482858487 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8126158565376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69244100 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240946189603328 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1072825125257853 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56642639 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232773957793280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232773976405504f¡(ý'Ã÷O L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69233957 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1070976935940608 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1072825482858487 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8126158565376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69244100 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240946189603328 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1072825125257853 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56642639 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232773957793280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232773976405504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚240946063624704 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1079103111545344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1079102753081856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8172105831424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8126158565376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16736450 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16736163 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1079102753081856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56642639 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69244100 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125876596 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28949748 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12213585 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8113895280103 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _step‚574 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8125817141248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784513141400015e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8113554718829 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12213588 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚6314.975547343 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56642641 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8125817141248 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1079103111545344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8172213197824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1070976952979968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚6999 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1070976952979968 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1070976935940608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69233957 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1¾WßFK_Ä‚(:Ô õ„öÒ©É°Ë network.sent‚ 1.1086364e+07 disk.rbd0.in‚0# disk.dm-0.out‚47918.70361328125 cpu‚25.669592954363335) disk./.usagePercent‚76.00466341988657% disk./.usageGB‚1166.8615951538086' disk.nvme0c0n1.out‚36630.8544921875 disk.nvme0n1.in‚ 0.00390625% disk.nvme1n1.out‚21913.2919921875 disk.dm-1.in‚0( proc.memory.availableMB‚ 493.08203125 proc.cpu.threads‚17 network.recv‚ 1.686374e+06 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚21913.2919921875 disk.md0.out‚ 190.41015625 disk.dm-0.in‚ 0.00390625# disk.dm-1.out‚10218.35205078125 disk.rbd0.out‚0% disk.nvme0n1.out‚36630.8544921875 disk.nvme1n1.in‚0 disk.md0.in‚0$ memory_percent‚96.99046611785889 proc.memory.rssMB‚3013.5( proc.memory.percent‚18.3929443359375s1–.žÅ‚(:“ õ„öÒÜ–óË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚125.873' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.95766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xÿ­2Æ- _wandbruntime‚6315  _runtime‚6315k`ÇÇýM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8114663107081 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28953837 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125904204 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1072973664313577 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56650434 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12215319 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56650432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232807156675584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1079252496153600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125904204 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8126927240192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1064859001206496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56650432 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1079252137682944 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1071125210442752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16738518 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1071125227482112 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56650434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚6315.604696059 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8173218608128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12215322 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1071125227482112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28954120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1079252496153600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125894045 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8126927240192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240980486837248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272"é&'È‚N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1079252496153600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125904204 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8126927240192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1064859001206496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56650432 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1079252137682944 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1071125210442752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16738518 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1071125227482112 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56650434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚6315.604696059 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8173218608128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12215322 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1071125227482112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28954120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1079252496153600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125894045 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8126927240192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240980486837248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8114663107081 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28953837 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125904204 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1072973664313577 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56650434 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12215319 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56650432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232807156675584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159¿½SŽ2É- _wandbruntime‚6315  _runtime‚6315®w"Ë Êƒ 1loss/FBt2/assignment_loss‚0.05604635894104959 1loss/SRqqt1/detection_loss‚0.6391430297913705 1loss/SRqqt2/detection_loss‚0.6548053388894236 epoch‚17 % _timestamp‚1.7845131420300107e+09 _runtime‚6315.605336231 0loss/FRt2/assignment_loss‚0.7969136316243052 2loss/FBt1/assignment_loss‚0.016370886850588292 /loss/FRt2/detection_loss‚1.5872857519186037 .loss/FBt2/detection_loss‚0.516151618747972 0loss/FRt1/detection_loss‚0.45895842549027144 /loss/FBt1/detection_loss‚0.4382300014772756 %loss/total_loss‚7.813146475169972 0loss/FRt1/assignment_loss‚1.6260149285811163 2loss/SRqqt1/assignment_loss‚0.5206482139391997 2loss/SRqqt2/assignment_loss‚0.5025782889187967 trainer/global_step‚6999 _step‚576˜ØSàˈ 0loss/FRt2/assignment_loss‚0.7969136316243052 2loss/FBt1/assignment_loss‚0.016370886850588292 /loss/FRt2/detection_loss‚1.5872857519186037 .loss/FBt2/detection_loss‚0.516151618747972 0loss/FRt1/detection_loss‚0.45895842549027144 /loss/FBt1/detection_loss‚0.4382300014772756 %loss/total_loss‚7.813146475169972 0loss/FRt1/assignment_loss‚1.6260149285811163 2loss/SRqqt1/assignment_loss‚0.5206482139391997 2loss/SRqqt2/assignment_loss‚0.5025782889187967 trainer/global_step‚6999 _step‚576 1loss/FBt2/assignment_loss‚0.05604635894104959 1loss/SRqqt1/detection_loss‚0.6391430297913705 1loss/SRqqt2/detection_loss‚0.6548053388894236 epoch‚17 % _timestamp‚1.7845131420300107e+09 _runtime‚6315.605336231Àü$›\Ì‚(:Ñ „…öÒ‹ßÖË proc.memory.rssMB‚3013.5 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 1.691373e+06 disk.nvme1n1.in‚0 disk.md0.in‚0# disk.dm-0.out‚47919.32470703125' disk.nvme0c0n1.out‚36631.9521484375 disk.md0.out‚ 190.97265625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 cpu‚25.669819856155026 proc.cpu.threads‚17 network.sent‚ 1.1136925e+07$ disk./.usageGB‚1166.861557006836 disk.dm-1.in‚0" disk.dm-1.out‚10218.5048828125 disk.nvme0c0n1.in‚ 0.00390625( proc.memory.percent‚18.3929443359375) disk./.usagePercent‚76.00466093514645& disk.nvme1n1.out‚21914.16357421875 disk.dm-0.in‚ 0.00390625% disk.nvme0n1.out‚36631.9521484375( disk.nvme1c1n1.out‚21914.16357421875$ memory_percent‚96.98925018310547% proc.memory.availableMB‚ 493.28125/él՞͂(:“ „…öÒ÷ž˜Ì gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.949' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.64966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÔé^΂(:Ó “…öÒ°Ø»Ë& disk.nvme1n1.out‚21918.18603515625 disk.md0.in‚0 proc.memory.rssMB‚3013.5 cpu‚25.670757427319817( disk.nvme0c0n1.out‚36634.40087890625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0" disk.dm-0.out‚47921.3720703125% disk./.usageGB‚1166.8617134094238 disk.rbd0.out‚0( disk.nvme1c1n1.out‚21918.18603515625 disk.md0.out‚ 191.5546875# memory_percent‚96.9907283782959 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0" disk.dm-1.out‚10221.7041015625' proc.memory.availableMB‚ 493.0390625( proc.memory.percent‚18.3929443359375 proc.cpu.threads‚17 network.sent‚ 1.1177963e+07 network.recv‚ 1.696244e+06) disk./.usagePercent‚76.00467112258093& disk.nvme0n1.out‚36634.40087890625-Zs+“Ï‚(:ˆ “…öÒÑ¢ÀË gpu.0.gpu‚75.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚222.036' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚74.012 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·¼ÇKЂ(:À ¢…öÒ÷å·Ë' proc.memory.availableMB‚ 493.2890625 proc.memory.rssMB‚3013.5 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625% disk./.usageGB‚1166.8617668151855% disk.nvme0n1.out‚36635.3876953125# disk.nvme1c1n1.out‚ 21919.015625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625 cpu‚25.67161064380152 network.sent‚ 1.1180986e+07 disk.md0.out‚ 192.06640625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚36635.3876953125( proc.memory.percent‚18.3929443359375 disk.md0.in‚0" disk.dm-0.out‚47921.9345703125 network.recv‚ 1.699508e+06( disk./.usagePercent‚76.0046746012171 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0! disk.nvme1n1.out‚ 21919.015625 disk.rbd0.out‚0 disk.nvme1n1.in‚0 disk.dm-1.out‚ 10221.875$ memory_percent‚96.98920249938965·z&žÑ‚(:“ ¢…öÒÜÝÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.587' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.19566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥¡ 2Ò- _wandbruntime‚6360  _runtime‚6360¯NÊ”ÈÓ ' lr-AdamW/pg1‚0.00045590788064600536 ' lr-AdamW/pg2‚0.00045590788064600536 trainer/global_step‚7049 $ _timestamp‚1.784513186674437e+09 _runtime‚6360.250048502 _step‚577“‡ËÍÔÇ ' lr-AdamW/pg1‚0.00045590788064600536 ' lr-AdamW/pg2‚0.00045590788064600536 trainer/global_step‚7049 $ _timestamp‚1.784513186674437e+09 _runtime‚6360.250048502 _step‚577Á^`c2Õ- _wandbruntime‚6360  _runtime‚6360,õ}!ÖòO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845131866755075e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1086572047290880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚242661024634368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69718609 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126750898 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1078390678087168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1078390661047808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8181386243072 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1078390661047808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29151322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8181727667200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234432971297280 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1080252499445495 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16851315 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚242660894461440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234432989909504 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1072083120951984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57032291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69728752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8227923164160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126761043 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57032289 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _runtime‚ 6360.25407429 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1086572405754368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57032291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69728752 PCDeviceStatsMonitor.on_train_batc[]ä{h_start/allocation.small_pool.freed‚69718609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12300288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚578 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16851037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚7049 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8181727667200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8228034724864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3qì =ý'×÷O aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1078390678087168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126761043 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57032289 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _runtime‚ 6360.25407429 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1086572405754368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57032291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69728752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69718609 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12300288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚578 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16851037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚7049 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8181727667200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8228034724864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845131866755075e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1086572047290880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚242661024634368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69718609 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126750898 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1078390678087168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1078390661047808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8181386243072 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1078390661047808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29151322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8181727667200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚234432971297280 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1080252499445495 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16851315 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚242660894461440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234432989909504 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1072083120951984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57032291 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69728752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8227923164160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8182496342016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1086721431891968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8182837773312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1086721790362624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16853362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1072230551619456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8170486881875 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69738422 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69728265 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29155673 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234466151567360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126778506 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57040082 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16853651 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29155381 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1086721790362624 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1080401038501331 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8182837773312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12302019 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8229041011712 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1080400680900585 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚242695312259072 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126778506 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8182496342016 % _timestamp‚1.7845131873049493e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1078538935549952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1078538935549952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8182496342016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1086721431891968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8182837773312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1086721790362624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16853362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1072230551619456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8170486881875 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69738422 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69728265 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29155673 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234466151567360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126778506 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57040082 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16853651 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29155381 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1086721790362624 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1080401038501331 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8182837773312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12302019 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8229041011712 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1080400680900585 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚242695312259072 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126778506 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8182496342016 % _timestamp‚1.7845131873049493e+09èã¶62Û- _wandbruntime‚6360  _runtime‚6360ÝV{¼è܃ /loss/FRt1/assignment_loss‚1.643085441103688 1loss/SRqqt1/detection_loss‚0.6431404299289438 0loss/FBt1/detection_loss‚0.46051600654196145 $loss/total_loss‚7.70463718490979 epoch‚17 2loss/FBt1/assignment_loss‚0.002240431352331503 /loss/FRt1/detection_loss‚0.5262590171771424 .loss/FRt2/detection_loss‚1.588229313263327 1loss/SRqqt2/detection_loss‚0.6567468535262168 _runtime‚6360.881724678 4loss/FBt2/assignment_loss‚0.00014651045860301082 % _timestamp‚1.7845131873061802e+09 _step‚580 0loss/FRt2/assignment_loss‚0.7547588284887355 2loss/SRqqt1/assignment_loss‚0.4879339321552289 2loss/SRqqt2/assignment_loss‚0.4917920895334974 .loss/FBt2/detection_loss‚0.449788331380Þ"¸ˆ!114 trainer/global_step‚7049ñÜ݈ $loss/total_loss‚7.70463718490979 epoch‚17 2loss/FBt1/assignment_loss‚0.002240431352331503 /loss/FRt1/detection_loss‚0.5262590171771424 .loss/FRt2/detection_loss‚1.588229313263327 1loss/SRqqt2/detection_loss‚0.6567468535262168 _runtime‚6360.881724678 4loss/FBt2/assignment_loss‚0.00014651045860301082 % _timestamp‚1.7845131873061802e+09 _step‚580 0loss/FRt2/assignment_loss‚0.7547588284887355 2loss/SRqqt1/assignment_loss‚0.4879339321552289 2loss/SRqqt2/assignment_loss‚0.4917920895334974 .loss/FBt2/detection_loss‚0.449788331380114 trainer/global_step‚7049 /loss/FRt1/assignment_loss‚1.643085441103688 1loss/SRqqt1/detection_loss‚0.6431404299289438 0loss/FBt1/detection_loss‚0.46051600654196145ÄÉ§Ê MÞ‚(: ±…öÒ͉Ì disk.rbd0.out‚0( disk.nvme0c0n1.out‚36637.96728515625 disk.nvme1c1n1.in‚0 disk.md0.in‚0$ memory_percent‚96.99115753173828 network.recv‚ 1.703113e+06) disk./.usagePercent‚76.00467758290525 disk.dm-0.in‚ 0.00390625% disk.nvme1n1.out‚21919.7861328125 disk.md0.out‚192.375 proc.memory.rssMB‚3013.5( proc.memory.percent‚18.3929443359375 proc.cpu.threads‚17% disk./.usageGB‚1166.8618125915527 disk.dm-1.in‚0 disk.dm-1.out‚10222 disk.rbd0.in‚0& disk.nvme0n1.out‚36637.96728515625% proc.memory.availableMB‚ 492.96875 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚21919.7861328125 disk.nvme1n1.in‚0 cpu‚25.672493328261016 network.sent‚ 1.1199684e+07# disk.dm-0.out‚47924.50927734375¨ŠMâ’ß‚(:‡ ±…öÒó£ÇÌ gpu.0.gpu‚11.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚284.544' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞgÍXà‚(:Í À…öÒÎþ²Ë network.sent‚ 1.1250034e+07& disk.nvme1n1.out‚21920.49169921875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625( proc.memory.percent‚18.3929443359375 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚21920.49169921875# disk.dm-0.out‚47925.11083984375 disk.dm-1.in‚0" disk.dm-1.out‚10222.1962890625$ memory_percent‚96.99256420135498 proc.cpu.threads‚17 network.recv‚ 1.707984e+06" disk.nvme0n1.out‚ 36638.8984375 disk.nvme1c1n1.in‚0 disk.md0.out‚ 192.765625 disk.rbd0.in‚0( proc.memory.availableMB‚ 492.73828125 proc.memory.rssMB‚3013.5) disk./.usagePercent‚76.00463260910911% disk./.usageGB‚1166.8611221313477 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625$ disk.nvme0c0n1.out‚ 36638.8984375 cpu‚25.673382935342183GäÕÎá‚(:’ À…öÒöóË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚166.419' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚55.473000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Aøaœ[â‚(:Ð Ï…öÒ¥®ÆË$ memory_percent‚96.99256420135498 proc.memory.rssMB‚3013.5% disk./.usageGB‚1166.8613243103027 disk.dm-1.in‚0 disk.rbd0.out‚0 cpu‚25.67414465763718 proc.cpu.threads‚17 network.sent‚ 1.1252956e+07 network.recv‚ 1.711389e+06) disk./.usagePercent‚76.00464577823173 disk.md0.in‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚21921.12353515625 disk.nvme1n1.in‚0( proc.memory.percent‚18.3929443359375 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚47927.02978515625# disk.nvme0n1.out‚36641.04296875& disk.nvme1n1.out‚21921.12353515625( proc.memory.availableMB‚ 492.73828125 disk.md0.out‚ 193.14453125" disk.dm-1.out‚10222.2509765625 disk.rbd0.in‚0% disk.nvme0c0n1.out‚36641.04296875 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0ŠÅw:žã‚(:“ Ï…öÒö¸–Ì gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.131' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.37699999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žƒ›d2ä- _wandbruntime‚6405  _runtime‚6405Ð,ÅÇåÁ & lr-AdamW/pg1‚0.0004537482218372255 & lr-AdamW/pg2‚0.0004537482218372255 trainer/global_step‚7099 % _timestamp‚1.7845132315370264e+09 _runtime‚6405.112780544 _step‚581ßô·~ÌæÆ _runtime‚6405.112780544 _step‚581 & lr-AdamW/pg1‚0.0004537482218372255 & lr-AdamW/pg2‚0.0004537482218372255 trainer/global_step‚7099 % _timestamp‚1.7845132315370264e+09Åêô.2ç- _wandbruntime‚6405  _runtime‚6405Ž>çV÷'èñO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70203261 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1094041341499904 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244375688592896 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1079454654325584 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70213404 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1087679158431869 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 115755008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8237296769024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8236955344896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1085804403194368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127635345 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244375822960128 trainer/global_step‚7099 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57421939 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12386985 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _runtime‚ 6405.1170909 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16965398 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236092003413504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1094041699963392 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1079454637286224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8224861706919 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57421939 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236091984801280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127625200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534ãÜKü'éöO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1094041699963392 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1079454637286224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8224861706919 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57421939 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚236091984801280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127625200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70203261 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1094041341499904 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244375688592896 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1079454654325584 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70213404 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1087679158431869 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 115755008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8237296769024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8236955344896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1085804403194368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127635345 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244375822960128 trainer/global_step‚7099 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57421939 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12386985 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _runtime‚ 6405.1170909 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16965398 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚236092003413504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0ÆhNP2ê- _wandbruntime‚6405  _runtime‚6405Œº”xëýM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8238065443840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57429732 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244410110339072 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1087827697487593 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127652808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1087828055088339 trainer/global_step‚7099 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236125183683584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8225629533897 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70212917 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8238065443840 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29356163 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1085952677696512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70223074 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70212917 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1085952677696512 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57429732 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8284926655488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1696744)-–‹ 4 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1085952660657152 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 % _timestamp‚1.7845132321669786e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57429734 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8238406875136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127642649 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1094191084571648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1094191084571648 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1094190726100992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚236125165071360 _step‚583 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8284819296256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656XÒ·Ž'ì‚N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8225629533897 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70212917 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8238065443840 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29356163 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1085952677696512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70223074 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70212917 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1085952677696512 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57429732 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8284926655488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16967444 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1085952660657152 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 % _timestamp‚1.7845132321669786e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57429734 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8238406875136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127642649 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1094191084571648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1094191084571648 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1094190726100992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚236125165071360 _step‚583 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8284819296256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8238065443840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57429732 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244410110339072 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1087827697487593 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127652808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1087828055088339 trainer/global_step‚7099 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236125183683584Ç“Á$J2í- _wandbruntime‚6405  _runtime‚6405 †5Ùî‚ 1loss/SRqqt1/detection_loss‚0.6146578529816772 /loss/FBt1/detection_loss‚0.5094667727662773 trainer/global_step‚7099 _runtime‚6405.743511025 2loss/SRqqt1/assignment_loss‚0.5284879273969006 2loss/SRqqt2/assignment_loss‚0.5246130831135072 3loss/FBt1/assignment_loss‚0.0014727894741748677 3loss/FBt2/assignment_loss‚0.0006735355903196528 0loss/FRt1/detection_loss‚0.44161300198432357 #loss/total_loss‚8.1263946287235 % _timestamp‚1.7845132321680696e+09 0loss/FRt1/assignment_loss‚1.6213684494428018 .loss/FRt2/detection_loss‚1.749200488673523 epoch‚17 0loss/FRt2/assignment_loss‚0.8976950614334462 /loss/SRqqt2/detection_loss‚0.71412934845015 /loss/FBt2/detection_loss‚0.5230163174163964 _step‚584³ð« ï‡ 2loss/SRqqt1/assignment_loss‚0.5284879273969006 2loss/SRqqt2/assignment_loss‚0.5246130831135072 3loss/FBt1/assignment_loss‚0.0014727894741748677 3loss/FBt2/assignment_loss‚0.0006735355903196528 0loss/FRt1/detection_loss‚0.44161300198432357 #loss/total_loss‚8.1263946287235 % _timestamp‚1.7845132321680696e+09 0loss/FRt1/assignment_loss‚1.6213684494428018 .loss/FRt2/detection_loss‚1.749200488673523 epoch‚17 0loss/FRt2/assignment_loss‚0.8976950614334462 /loss/SRqqt2/detection_loss‚0.71412934845015 /loss/FBt2/detection_loss‚0.5230163174163964 _step‚584 1loss/SRqqt1/detection_loss‚0.6146578529816772 /loss/FBt1/detection_loss‚0.5094667727662773 trainer/global_step‚7099 _runtime‚6405.743511025ÈPºýZð‚(:Ï Þ…öÒ»ÌóË( proc.memory.percent‚18.3929443359375 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme0n1.out‚36642.1533203125' proc.memory.availableMB‚ 492.3828125 cpu‚25.67501765068358 proc.cpu.threads‚17$ disk./.usageGB‚1166.861312866211 disk.md0.out‚ 193.55859375 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚21921.8623046875% disk.nvme1n1.out‚21921.8623046875 network.recv‚ 1.714491e+06( disk./.usagePercent‚76.0046450328097# disk.dm-1.out‚10222.41064453125 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚96.9947338104248 proc.memory.rssMB‚3013.5 network.sent‚ 1.1271676e+07 disk.md0.in‚0# disk.dm-0.out‚47927.81884765625' disk.nvme0c0n1.out‚36642.1533203125 disk.nvme1n1.in‚0í¿žñ‚(:“ Þ…öÒª¹Ì gpu.0.gpu‚47.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.313' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.43766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Usu`]ò‚(:Ò í…öÒÒ´ÝË disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚36644.21533203125$ memory_percent‚96.99432849884033 network.sent‚ 1.1322304e+07# disk.dm-1.out‚10222.45751953125( proc.memory.availableMB‚ 492.44921875( proc.memory.percent‚18.3929443359375 cpu‚25.675929728173067 proc.cpu.threads‚17% disk./.usageGB‚1166.8620796203613' disk.nvme1c1n1.out‚21923.0400390625 disk.nvme0n1.in‚ 0.00390625 proc.memory.rssMB‚3013.5 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚21923.0400390625 disk.md0.in‚0 disk.md0.out‚ 193.98046875 disk.dm-0.out‚ 47930.1171875 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚36644.21533203125 network.recv‚ 1.720143e+06) disk./.usagePercent‚76.00469497608606 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625"÷ÂœŸó‚(:” í…öÒŸ ¡Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚120.764' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.254666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛþË(^ô‚(:Ó ü…öÒÇÔ¢Ë& disk.nvme0n1.out‚36644.83642578125 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 network.sent‚ 1.1325268e+07% disk./.usageGB‚1166.8619499206543 disk.nvme1n1.in‚0% disk.nvme1n1.out‚21923.3662109375$ memory_percent‚96.99432849884033 proc.memory.rssMB‚3013.5( proc.memory.percent‚18.3929443359375 proc.cpu.threads‚17) disk./.usagePercent‚76.00468652796967 disk.md0.in‚0 disk.md0.out‚ 194.07421875 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚47930.73828125# disk.dm-1.out‚10222.55712890625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚36644.83642578125 network.recv‚ 1.723342e+06 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚21923.3662109375( proc.memory.availableMB‚ 492.44921875 cpu‚25.676723983070605*+„7“õ‚(:ˆ ü…öÒ‰–òË gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚265.554' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tPQ’2ö- _wandbruntime‚6450  _runtime‚6450Lû6Ç÷Á trainer/global_step‚7149 % _timestamp‚1.7845132769020066e+09 _runtime‚6450.477556314 _step‚585 & lr-AdamW/pg1‚0.0004515791533817047 & lr-AdamW/pg2‚0.0004515791533817047aÊL»ÌøÆ trainer/global_step‚7149 % _timestamp‚1.7845132769020066e+09 _runtime‚6450.477556314 _step‚585 & lr-AdamW/pg1‚0.0004515791533817047 & lr-AdamW/pg2‚0.0004515791533817047É‘üÜ)2ù- _wandbruntime‚6450  _runtime‚6450ª\‘ø'úòO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29553800 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29553529 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128499502 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57811589 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70687913 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1093218111262208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12473688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1095106175018877 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845132769026983e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1093218128301568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70687913 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 trainer/global_step‚7149 _step‚586 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1093218111262208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70698056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1086826170659824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8292865870848 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237751016917504 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1101510635708928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57811591 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1101510994172416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8292524446720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1101510635708928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57811589 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29553529 \ODeviceStatsMonitor.on_traiê‹‚¨"n_batch_start/inactive_split_bytes.small_pool.current‚98977792 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128499502 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57811589 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70687913 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1093218111262208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12473688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1095106175018877 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845132769026983e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1093218128301568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70687913 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 trainer/global_step‚7149 _step‚586 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1093218111262208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70698056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1086826170659824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8292865870848 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237751016917504 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1101510635708928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57811591 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1101510994172416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8292524446720 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1101510635708928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57811589 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1093366402803712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128516951 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8293634545664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57819382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8281112747305 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _step‚587 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70697569 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚246124818566144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8293975976960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29557769 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12475422 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚246124936148992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845132775291333e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1093366385764352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128527110 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1101660378780672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128527110 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57819384 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8340639990784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8281453308691 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70697569 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1101660020310016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1095254714074601 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1086973618366656 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1086973601327296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1093366402803712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1093366385764352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12475419 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237784178575360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚6451.104857228 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128516951 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29557493 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57819382 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8293975976960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29557769 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12475422 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚246124936148992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845132775291333e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1093366385764352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128527110 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1101660378780672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128527110 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57819384 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8340639990784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8281453308691 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70697569 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1101660020310016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1095254714074601 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1086973618366656 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1086973601327296 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1093366402803712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1093366385764352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12475419 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237784178575360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚6451.104857228 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128516951 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29557493 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57819382 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1093366402803712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128516951 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8293634545664Ë©Ž®2ÿ- _wandbruntime‚6451  _runtime‚6451ßók¬ €† epoch‚17 3loss/SRqqt1/assignment_loss‚0.49555347818576745 2loss/FBt1/assignment_loss‚0.012671257033072376 /loss/FRt2/detection_loss‚1.7669417936115075 1loss/SRqqt2/detection_loss‚0.8026250309362971 /loss/FBt1/detection_loss‚0.4079135895638804 % _timestamp‚1.7845132775303762e+09 _runtime‚6451.105791429 0loss/FRt2/assignment_loss‚0.8926377797911981 trainer/global_step‚7149 _step‚588 %loss/total_loss‚8.078447691712611 3loss/SRqqt2/assignment_loss‚0.49375611145649895 /loss/FRt1/detection_loss‚0.4955988112736792 1loss/SRqqt1/detection_loss‚0.6717297418291265 /loss/FRt1/assignment_loss‚1.605213814093549 3loss/FBt2/assignment_loss‚6.807766259244163e-05 /loss/FBt2/detection_loss‚0.4337382062754415üà‹ %loss/total_loss‚8.078447691712611 3loss/SRqqt2/assignment_loss‚0.49375611145649895 /loss/FRt1/detection_loss‚0.4955988112736792 1loss/SRqqt1/detection_loss‚0.6717297418291265 /loss/FRt1/assignment_loss‚1.605213814093549 3loss/FBt2/assignment_loss‚6.807766259244163e-05 /loss/FBt2/detection_loss‚0.4337382062754415 epoch‚17 3loss/SRqqt1/assignment_loss‚0.49555347818576745 2loss/FBt1/assignment_loss‚0.012671257033072376 /loss/FRt2/detection_loss‚1.7669417936115075 1loss/SRqqt2/detection_loss‚0.8026250309362971 /loss/FBt1/detection_loss‚0.4079135895638804 % _timestamp‚1.7845132775303762e+09 _runtime‚6451.105791429 0loss/FRt2/assignment_loss‚0.8926377797911981 trainer/global_step‚7149 _step‚588ÌSn®Y‚‚(:Î ‹†öÒµ÷„Ì network.sent‚ 1.1344e+07 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚36647.2548828125# disk.dm-0.out‚47933.24072265625 network.recv‚ 1.72689e+06) disk./.usagePercent‚76.00468454017756 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme1n1.out‚21924.2353515625' proc.memory.availableMB‚ 492.4140625 proc.memory.rssMB‚3013.5( proc.memory.percent‚18.3929443359375 disk.nvme1c1n1.in‚0% disk./.usageGB‚1166.8619194030762# disk.dm-1.out‚10222.63525390625' disk.nvme0c0n1.out‚36647.2548828125$ memory_percent‚96.99454307556152 cpu‚25.677620064099706 proc.cpu.threads‚17' disk.nvme1c1n1.out‚21924.2353515625 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 194.3984375 disk.dm-0.in‚ 0.00390625c3푃‚(:† ‹†öÒ¤¬ÂÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.616' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.872 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ‘–\„‚(:Ñ š†öÒÓÚ¦Ë) disk./.usagePercent‚75.82766767861186 disk.nvme1n1.in‚0 disk.md0.out‚ 194.6796875# disk.dm-1.out‚10222.71630859375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 network.recv‚ 1.731992e+06& disk.nvme1c1n1.out‚21924.736328125 disk.dm-1.in‚0( proc.memory.availableMB‚ 492.41796875( proc.memory.percent‚18.3929443359375 proc.cpu.threads‚17% disk.nvme0n1.out‚36650.6943359375 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 proc.memory.rssMB‚3013.5 cpu‚25.678409038885203% disk./.usageGB‚1164.1442680358887 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚21924.736328125# disk.dm-0.out‚47936.50244140625' disk.nvme0c0n1.out‚36650.6943359375$ memory_percent‚96.99451923370361 network.sent‚ 1.1392441e+078±¢†ž…‚(:“ š†öÒÞÐùË gpu.0.gpu‚91.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚269.192' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.73066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yÕ;t]†‚(:Ò ©†öÒ¥¸µË disk.rbd0.out‚0$ memory_percent‚96.99451923370361) disk./.usagePercent‚75.82766494539773 disk.md0.out‚ 195.2578125 disk.rbd0.in‚0 proc.memory.rssMB‚3013.5( proc.memory.percent‚18.3929443359375 proc.cpu.threads‚17 network.sent‚ 1.1399157e+07% disk./.usageGB‚1164.1442260742188 disk.md0.in‚0! disk.dm-0.out‚47947.185546875 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚36660.61279296875 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚21926.7880859375 disk.dm-1.in‚0( proc.memory.availableMB‚ 492.41796875 cpu‚25.67922573103177 network.recv‚ 1.735355e+06& disk.nvme0n1.out‚36660.61279296875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚21926.7880859375 disk.dm-0.in‚ 0.00390625# disk.dm-1.out‚10222.76708984375Å,ÄØž‡‚(:“ ©†öÒ¹ÊøË gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚264.788' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.26266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ@¶&2ˆ- _wandbruntime‚6495  _runtime‚6495O;6ljÁ & lr-AdamW/pg1‚0.0004494008145946567 & lr-AdamW/pg2‚0.0004494008145946567 trainer/global_step‚7199 % _timestamp‚1.7845133216270905e+09 _runtime‚6495.202738411 _step‚589`®lLÌŠÆ % _timestamp‚1.7845133216270905e+09 _runtime‚6495.202738411 _step‚589 & lr-AdamW/pg1‚0.0004494008145946567 & lr-AdamW/pg2‚0.0004494008145946567 trainer/global_step‚7199ÍŠ7/l2‹- _wandbruntime‚6495  _runtime‚6495¡Ž <.ç3eù'ŒóO ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1108980288381440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8348434972672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247805453068288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1100631853408768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17194023 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1102533549206519 % _timestamp‚1.7845133216282258e+09 _runtime‚6495.203824653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1108979929917952 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1100631853408768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1102533191605885 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1108980288381440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58201241 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8348093548544 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58201239 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8395321571840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚590Éúƒƒþ'øO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58201241 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8348093548544 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58201239 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8395321571840 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚590 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1108980288381440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8348434972672 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247805453068288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1100631853408768 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17194023 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1102533549206519 % _timestamp‚1.7845133216282258e+09 _runtime‚6495.203824653 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1108979929917952 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1100631853408768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1102533191605885 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1108980288381440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374ή„³2Ž- _wandbruntime‚6495  _runtime‚6495Ñèû@'ýM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71192378 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247839753423360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8396433275392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1100780127910912 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1109129672989696 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1094345134700896 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 trainer/global_step‚7199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71182221 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12562122 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1094345151740256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8336595960713 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12562119 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29758062 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247839625354752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58209034 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58209032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17195943 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129401412 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129391253 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129391253 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129401412 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8396542731776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1102682088262355 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239443210691584 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8349203647488 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17196220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚239443192079360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1100780110871552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8349545078784 % _timestamp‚1.7845133222551014e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8349203647488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6495.830841579 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¤3N=±‚N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129401412 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8396542731776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1102682088262355 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239443210691584 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8349203647488 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17196220 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚239443192079360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1100780110871552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8349545078784 % _timeä„ ~Wstamp‚1.7845133222551014e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8349203647488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚6495.830841579 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71192378 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247839753423360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8396433275392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1100780127910912 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1109129672989696 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1094345134700896 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 trainer/global_step‚7199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71182221 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12562122 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1094345151740256 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8336595960713 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12562119 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29758062 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247839625354752 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58209034 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58209032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17195943 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129401412 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129391253 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129391253 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ÏŽÞ¶2‘- _wandbruntime‚6495  _runtime‚6495uç ’… /loss/FBt1/detection_loss‚0.4075952248407923 epoch‚17 2loss/FBt2/assignment_loss‚0.003967723644625028 /loss/FRt2/detection_loss‚1.9688961082404661 1loss/SRqqt2/detection_loss‚0.7349511761586239 trainer/global_step‚7199 1loss/SRqqt1/detection_loss‚0.5366255685049924 % _timestamp‚1.7845133222561865e+09 _runtime‚ 6495.83148794 /loss/FBt2/detection_loss‚0.5064653788316757 %loss/total_loss‚7.991804552230803 0loss/FRt1/assignment_loss‚1.4955805161925317 0loss/FRt2/assignment_loss‚0.8605883856012364 3loss/SRqqt1/assignment_loss‚0.49351388601607515 2loss/SRqqt2/assignment_loss‚0.5222340233806868 0loss/FRt1/detection_loss‚0.45544215885264844 _step‚592 2loss/FBt1/assignment_loss‚0.005944401966449395d\Ë“Š /loss/FBt2/detection_loss‚0.5064653788316757 %loss/total_loss‚7.991804552230803 0loss/FRt1/assignment_loss‚1.4955805161925317 0loss/FRt2/assignment_loss‚0.8605883856012364 3loss/SRqqt1/assignment_loss‚0.49351388601607515 2loss/SRqqt2/assignment_loss‚0.5222340233806868 0loss/FRt1/detection_loss‚0.45544215885264844 _step‚592 2loss/FBt1/assignment_loss‚0.005944401966449395 /loss/FBt1/detection_loss‚0.4075952248407923 epoch‚17 2loss/FBt2/assignment_loss‚0.003967723644625028 /loss/FRt2/detection_loss‚1.9688961082404661 1loss/SRqqt2/detection_loss‚0.7349511761586239 trainer/global_step‚7199 1loss/SRqqt1/detection_loss‚0.5366255685049924 % _timestamp‚1.7845133222561865e+09 _runtime‚ 6495.83148794ÐPð]”‚(:Ò ¸†öÒÉòëË network.sent‚ 1.1414926e+07& disk.nvme0c0n1.out‚36662.205078125 disk.nvme0n1.in‚ 0.00390625$ disk.nvme0n1.out‚36662.205078125( disk.nvme1c1n1.out‚21927.39111328125 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625% disk./.usageGB‚1164.1448860168457 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0# memory_percent‚96.9951868057251( proc.memory.availableMB‚ 492.30859375 cpu‚25.685169273687748 proc.cpu.threads‚17) disk./.usagePercent‚75.82770793140176 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.percent‚18.3929443359375 network.recv‚ 1.738352e+06& disk.nvme1n1.out‚21927.39111328125 disk.md0.out‚ 195.60546875# disk.dm-0.out‚47948.49072265625 disk.dm-1.in‚0" disk.dm-1.out‚10222.9111328125 proc.memory.rssMB‚3013.5 disk.nvme1n1.in‚0«Ç0Ìœ•‚(:‘ ¸†öÒ¨ê¸Ì gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚199.423' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.47433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì¡=2–- _wandbruntime‚6512  _runtime‚6512‘5ìÅY=—Óz @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 'particle/accuracy‚0.855557918548584 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 )loss/val_total_loss‚8.140493130871286 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7068608999252319 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05144824832677841 particle/accuracy_1_of_2‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6798048615455627 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6260262131690979 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2292143702507019 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02722916752099991 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5092007517814636 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2223573625087738 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03896241635084152 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19879311323165894 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8979024291038513 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6496188640594482 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4999504089355469 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32754865288734436 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22961227595806122 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9012848734855652 )jet/accuracy_1_of_4‚0.981631875038147 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7153817415237427 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.761001467704773 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7629010081291199 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8814069628715515 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21420669555664062 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5852961540222168 'particle/f_score‚0.6581677198410034 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7249561548233032 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5203181505203247 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.27865076065063477 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.017014874145388603 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8612223267555237 +particle/specificity‚0.9124639630317688 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8228458166122437 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.1627991646528244 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21548298001289368 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.4035901427268982 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6660982966423035 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902462959289551 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7405128479003906 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2275269329547882 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9027210474014282 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04952393099665642 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_4‚0.10299699753522873 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6385480165481567 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6923102140426636 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 jet/accuracy_1_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7056203484535217 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22658412158489227 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963371753692627 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6872074604034424 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.875 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7170900702476501 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 particle/accuracy_5_of_5‚1 % _timestamp‚1.7845133394148428e+09 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7519704103469849 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6594215631484985 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1875000149011612 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.40752705931663513 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17555932700634003 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21041765809059143 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7220219373703003 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7"Purity/*FRt0SRqqt1FBt/event_purity‚0.90678471326828 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.789235532283783 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9621760249137878 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16342610120773315 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7878719568252563 +jet/accuracy_3_of_3‚0.08222882449626923 particle/accuracy_1_of_4‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.453125 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21420669555664062 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.031927019357681274 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968475103378296 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.49454325437545776 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.875 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.9027210474014282 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_3_of_5‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7572407126426697 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8207793235778809 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19054608047008514 particle/accuracy_2_of_2‚1 _runtime‚6512.990954336 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28955933451652527 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9899393916130066 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8612223267555237 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6619226336479187 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941513538360596 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1313730925321579 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1830717921257019 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 epoch‚17 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5736064910888672 jet/accuracy_3_of_5‚1 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.046538904309272766 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03661298379302025 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7785722017288208 +particle/sensitivity‚0.6475088596343994 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1853449046611786 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7003719210624695 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12151719629764557 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_3_of_3‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3409193754196167 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02275734394788742 *jet/accuracy_1_of_3‚0.9340366125106812 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7456304430961609 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7671636939048767 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14771100878715515 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2021816372871399 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8208063244819641 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7109375596046448 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6821509599685669 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6845276951789856 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04701939970254898 /particle/accuracy_4_of_4‚0.9505154490470886 7validation_average_jet_accuracy‚0.38112562894821167 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6837476491928101 *jet/accuracy_1_of_2‚0.8091772198677063 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚593 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.199164479970932 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8890399932861328 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13716334104537964 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02275734394788742 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2063586562871933 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6845276951789856 +jet/accuracy_1_of_1‚0.33607855439186096 particle/accuracy_2_of_5‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5723764896392822 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6674466133117676 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46252718567848206 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2514793872833252 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 trainer/global_step‚7199 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7672876715660095 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03869549557566643 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18563802540302277 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2273237109184265 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47050172090530396 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7733599543571472 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5737499594688416 *jet/accuracy_2_of_4‚0.8091715574264526 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.758019208908081 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6763904690742493 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8990613222122192†Ò♘Øz 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47050172090530396 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7733599543571472 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5737499594688416 *jet/accuracy_2_of_4‚0.8091715574264526 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.758019208908081 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6763904690742493 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8990613222122192 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 'particle/accuracy‚0.855557918548584 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 )loss/val_total_loss‚8.140493130871286 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7068608999252319 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05144824832677841 particle/accuracy_1_of_2‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6798048615455627 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6260262131690979 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2292143702507019 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02722916752099991 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5092007517814636 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2223573625087738 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03896241635084152 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19879311323165894 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8979024291038513 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6496188640594482 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4999504089355469 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32754865288734436 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22961227595806122 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9012848734855652 )jet/accuracy_1_of_4‚0.981631875038147 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7153817415237427 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.761001467704773 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7629010081291199 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8814069628715515 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21420669555664062 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5852961540222168 'particle/f_score‚0.6581677198410034 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7249561548233032 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5203181505203247 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.27865076065063477 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.017014874145388603 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8612223267555237 +particle/specificity‚0.9124639630317688 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8228458166122437 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.1627991646528244 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21548298001289368 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.4035901427268982 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6660982966423035 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902462959289551 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7405128479003906 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2275269329547882 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9027210474014282 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04952393099665642 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_4‚0.10299699753522873 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6385480165481567 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6923102140426636 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 jet/accuracy_1_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7056203484535217 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22658412158489227 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963371753692627 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6872074604034424 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.875 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7170900702476501 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 particle/accuracy_5_of_5‚os*Å$1 % _timestamp‚1.7845133394148428e+09 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7519704103469849 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6594215631484985 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1875000149011612 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.40752705931663513 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17555932700634003 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21041765809059143 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7220219373703003 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7"Purity/*FRt0SRqqt1FBt/event_purity‚0.90678471326828 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.789235532283783 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9621760249137878 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16342610120773315 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7878719568252563 +jet/accuracy_3_of_3‚0.08222882449626923 particle/accuracy_1_of_4‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.453125 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21420669555664062 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.031927019357681274 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968475103378296 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.49454325437545776 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.875 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.9027210474014282 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_3_of_5‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7572407126426697 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8207793235778809 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19054608047008514 particle/accuracy_2_of_2‚1 _runtime‚6512.990954336 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28955933451652527 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9899393916130066 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8612223267555237 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6619226336479187 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941513538360596 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1313730925321579 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1830717921257019 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 epoch‚17 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5736064910888672 jet/accuracy_3_of_5‚1 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.046538904309272766 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03661298379302025 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7785722017288208 +particle/sensitivity‚0.6475088596343994 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1853449046611786 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7003719210624695 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12151719629764557 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_3_of_3‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3409193754196167 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02275734394788742 *jet/accuracy_1_of_3‚0.9340366125106812 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7456304430961609 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7671636939048767 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14771100878715515 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2021816372871399 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8208063244819641 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7109375596046448 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6821509599685669 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6845276951789856 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04701939970254898 /particle/accuracy_4_of_4‚0.9505154490470886 7validation_average_jet_accuracy‚0.38112562894821167 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6837476491928101 *jet/accuracy_1_of_2‚0.8091772198677063 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚593 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.199164479970932 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8890399932861328 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13716334104537964 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02275734394788742 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2063586562871933 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6845276951789856 +jet/accuracy_1_of_1‚0.33607855439186096 particle/accuracy_2_of_5‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5723764896392822 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6674466133117676 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46252718567848206 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2514793872833252 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 trainer/global_step‚7199 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7672876715660095 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03869549557566643 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18563802540302277 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2273237109184265 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686Ñü®t^™‚(:Ó džöÒƒñçË disk.md0.out‚ 196.078125" disk.dm-1.out‚10468.3291015625 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.685318190406306% disk./.usageGB‚1164.1446380615234 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0" disk.dm-0.out‚47951.3681640625 disk.dm-1.in‚0) disk./.usagePercent‚75.82769178059101 disk.dm-0.in‚ 0.00390625# memory_percent‚95.4878568649292 proc.memory.rssMB‚3013.5 proc.cpu.threads‚17 network.recv‚ 1.743695e+06( disk.nvme0c0n1.out‚36665.27490234375& disk.nvme1n1.out‚22173.62646484375 disk.md0.in‚0( proc.memory.availableMB‚ 739.26953125( proc.memory.percent‚18.3929443359375 network.sent‚ 1.1477307e+07& disk.nvme0n1.out‚36665.27490234375( disk.nvme1c1n1.out‚22173.62646484375Ɣğš‚(:” džöÒšÆ«Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.264' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.087999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KàR›‚(:Ç Ö†öÒ¹ÉÔË) disk./.usagePercent‚76.00757305056435# disk.dm-1.out‚10532.43701171875' disk.nvme1c1n1.out‚22238.2978515625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚22238.2978515625 proc.memory.rssMB‚3013.5 network.sent‚ 1.1523417e+07$ disk./.usageGB‚1166.906265258789 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.out‚ 38579.9375$ memory_percent‚96.99394702911377 cpu‚25.684704774226024 disk.md0.out‚ 196.47265625 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme0n1.out‚ 38579.9375 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 492.51171875( proc.memory.percent‚18.3929443359375 disk.md0.in‚0# disk.dm-0.out‚49865.74072265625 proc.cpu.threads‚17 network.recv‚ 1.748558e+06¢0ëžœ‚(:“ Ö†öÒ‘ÑØË gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.637' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.54566666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ƒ¢\‚(:Ñ å†öÒ¸ÕË( disk./.usagePercent‚76.0075765292005& disk.nvme0c0n1.out‚38583.740234375# disk.dm-1.out‚10713.88623046875 disk.rbd0.out‚0( proc.memory.percent‚18.3929443359375 cpu‚25.684168971429607% disk./.usageGB‚1166.9063186645508 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme0n1.out‚38583.740234375 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚49869.1181640625 disk.dm-1.in‚0( proc.memory.availableMB‚ 493.01171875 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚22420.75048828125 disk.md0.in‚0 disk.md0.out‚ 197.15234375 proc.memory.rssMB‚3013.5 proc.cpu.threads‚17 network.sent‚ 1.1528214e+07 network.recv‚ 1.75216e+06 disk.nvme1n1.in‚0& disk.nvme1n1.out‚22420.75048828125 disk.rbd0.in‚0$ memory_percent‚96.99089527130127;®÷|‘ž‚(:† å†öÒáÌ gpu.0.gpu‚61.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚256.83' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚85.61 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^k×á2Ÿ- _wandbruntime‚6558  _runtime‚6558ÞƠÀ _step‚594 & lr-AdamW/pg1‚0.0004472133453867111 & lr-AdamW/pg2‚0.0004472133453867111 trainer/global_step‚7249 % _timestamp‚1.7845133846409504e+09 _runtime‚ 6558.21654363)ÑË¡Å % _timestamp‚1.7845133846409504e+09 _runtime‚ 6558.21654363 _step‚594 & lr-AdamW/pg1‚0.0004472133453867111 & lr-AdamW/pg2‚0.0004472133453867111 trainer/global_step‚7249Òý*í2¢- _wandbruntime‚6558  _runtime‚6558h®À\ø'£òO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8424321912832 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚249563592263168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1118412244304384 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1118411885840896 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1109987922391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58713699 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241091622587904 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1103490510454608 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1109987905352192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚249563478867456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1111902125650653 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1118412244304384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1118411885840896 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1109987905352192 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29995176 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452ÝGZ-"¤÷O VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1111901768050019 _step‚595 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8424321912832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8423980488704 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29994894 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12651201 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71738716 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1118412244304384 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1118411885840896 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1109987922391552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58713699 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241091622587904 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1103490510454608 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1109987905352192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚249563478867456 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1111902125650653 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1118412244304384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1118411885840896 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1109987905352192 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29995176 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8424321912832 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚249563592263168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1110136179854336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12652935 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1112050664706489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130480021 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241124784245760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚249597772848128 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1112050307105743 _step‚596 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1118561628912640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130469862 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71748372 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29999183 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8412383023023 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58721492 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1110136196893696 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130480021 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17346245 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17345973 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8412723584409 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29998908 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71758529 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1110136179854336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58721490 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8425090587648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8425432018944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0›®Ÿ'§N _runtime‚6558.846536687 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1110136179854336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12652935 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1112050664706489 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130480021 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241124784245760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚249597772848128 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1112050307105743 _step‚596 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1118561628912640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130469862 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71748372 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29999183 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8412383023023 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58721492 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1110136196893696 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130480021 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17346245 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17345973 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8412723584409 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29998908 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71758529 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1110136179854336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58721490 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8425090587648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8425432018944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1110136196893696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58721492 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚249597890430976 trainer/global_step‚7249ÔcÈ ‰2¨- _wandbruntime‚6558  _runtime‚6558TDSL ©„ 2loss/SRqqt1/assignment_loss‚0.4894756460671488 0loss/SRqqt1/detection_loss‚0.644627235507484 1loss/SRqqt2/detection_loss‚0.7869774565368475 .loss/FBt2/detection_loss‚0.552423553969129 $loss/total_loss‚8.07815615366907 trainer/global_step‚7249 0loss/FRt2/assignment_loss‚0.8413129486479201 3loss/FBt1/assignment_loss‚0.0007749285271115968 0loss/FRt1/detection_loss‚0.47327853701062206 % _timestamp‚1.7845133852718525e+09 _runtime‚6558.847347729 _step‚597 0loss/FRt1/assignment_loss‚1.5561292503200235 3loss/FBt2/assignment_loss‚0.0005521088287739673 /loss/FBt1/detection_loss‚0.4335071460203025 epoch‚18 2loss/SRqqt2/assignment_loss‚0.5149556754548209 /loss/FRt2/detection_loss‚1.7841416667788863Ðs.Tª‰ 2loss/SRqqt1/assignment_loss‚0.4894756460671488 0loss/SRqqt1/detection_loss‚0.644627235507484 1loss/SRqqt2/detection_loss‚0.7869774565368475 .loss/FBt2/detection_loss‚0.552423553969129 $loss/total_loss‚8.07815615366907 trainer/global_step‚7249 0loss/FRt2/assignment_loss‚0.8413129486479201 3loss/FBt1/assignment_loss‚0.0007749285271115968 0loss/FRt1/detection_loss‚0.47327853701062206 % _timestamp‚1.7845133852718525e+09 _runtime‚6558.847347729 _step‚597 0loss/FRt1/assignment_loss‚1.5561292503200235 3loss/FBt2/assignment_loss‚0.0005521088287739673 /loss/FBt1/detection_loss‚0.4335071460203025 epoch‚18 2loss/SRqqt2/assignment_loss‚0.5149556754548209 /loss/FRt2/detection_loss‚1.7841416667788863Õ“}"[«‚(:Ð ô†öÒñÀ²Ë disk.nvme0n1.in‚ 0.00390625 disk.dm-0.out‚ 50738.1328125# disk.dm-1.out‚10714.03564453125 disk.rbd0.out‚0' disk.nvme1c1n1.out‚22547.2880859375 disk.nvme1n1.in‚0% disk.nvme1n1.out‚22547.2880859375$ memory_percent‚96.99108600616455 disk.dm-0.in‚ 0.00390625 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 492.98046875 cpu‚25.683599927813855 proc.cpu.threads‚17 network.recv‚ 1.754541e+06) disk./.usagePercent‚76.00460378612375% disk./.usageGB‚1166.8606796264648 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 proc.memory.rssMB‚3013.5( proc.memory.percent‚18.3929443359375 network.sent‚ 1.1530219e+07 disk.dm-1.in‚0( disk.nvme0c0n1.out‚39327.13525390625& disk.nvme0n1.out‚39327.13525390625 disk.md0.in‚0 disk.md0.out‚ 197.5156257çÛ^œ¬‚(:‘ ô†öÒòÛ†Ì gpu.0.gpu‚89.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚275.9' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.96666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—º®&W­‚(:Ì ƒ‡öÒï¯ÛË network.sent‚ 1.1594476e+07 network.recv‚ 1.759758e+06# disk.dm-1.out‚10714.08642578125 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme0c0n1.out‚39336.68359375 disk.nvme0n1.in‚ 0.00390625# disk.nvme0n1.out‚39336.68359375 disk.nvme1n1.in‚0 disk.dm-1.in‚0# disk.nvme1n1.out‚22549.66796875 disk.md0.out‚ 197.83203125$ memory_percent‚96.99108600616455( proc.memory.availableMB‚ 492.98046875 proc.memory.rssMB‚3013.5( proc.memory.percent‚18.3929443359375 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625 cpu‚25.683066532412145) disk./.usagePercent‚76.00457272687228% disk./.usageGB‚1166.8602027893066 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚22549.66796875 disk.md0.in‚0# disk.dm-0.out‚50749.30517578125](á“®‚(:ˆ ƒ‡öÒÕÍ­Ì gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚218.706' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚72.902 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YÞ§S¯‚(:È ’‡öÒ÷ÇË disk.md0.in‚0 disk.rbd0.out‚0 network.recv‚ 1.763007e+06& disk.nvme0n1.out‚39337.84033203125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚39337.84033203125( proc.memory.availableMB‚ 492.73046875 proc.memory.rssMB‚3013.5 cpu‚25.682539352405175 disk.md0.out‚ 198.4140625 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 22550.640625 disk.nvme0c0n1.in‚ 0.00390625# memory_percent‚96.9926118850708( proc.memory.percent‚18.3929443359375% disk./.usageGB‚1166.3846321105957# disk.dm-0.out‚50750.00830078125 disk.dm-1.in‚0 disk.dm-1.out‚10714.28515625 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625 network.sent‚ 1.159744e+07) disk./.usagePercent‚75.97359596877354 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0# disk.nvme1c1n1.out‚ 22550.640625ÁOH›°‚(: ’‡öÒŽŸÊË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.37' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.45666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J¦C2±- _wandbruntime‚6602  _runtime‚6602ÜÙDzÁ & lr-AdamW/pg1‚0.0004450168862549278 & lr-AdamW/pg2‚0.0004450168862549278 trainer/global_step‚7299 % _timestamp‚1.7845134286193244e+09 _runtime‚6602.195872147 _step‚598œy„±Ì³Æ _step‚598 & lr-AdamW/pg1‚0.0004450168862549278 & lr-AdamW/pg2‚0.0004450168862549278 trainer/global_step‚7299 % _timestamp‚1.7845134286193244e+09 _runtime‚6602.195872147ÖÖæ œ2´- _wandbruntime‚6602  _runtime‚6602ð*¸CRµóO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1117401647498752 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8479891014656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8527713970688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59103349 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8479549590528 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1117401630459392 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72223368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12737904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1119328784637027 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30197398 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17459203 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚1[Ž'ܧ2303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131336860 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30197104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131326715 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1125881538513408 trainer/global_step‚7299 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1117401647498752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8466757848179 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12737901 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59103347 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8527819239936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251278455331840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251278331450368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚242750617479680 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72223368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59103349 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1110862026788848 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6602.195933217 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1117401630459392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1125881180049920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚242750636091904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845134286200407e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1125881180049920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1125881180049920 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17459203 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131336860 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30197104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131326715 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1125881538513408 trainer/global_step‚7299 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1117401647498752 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8466757848179 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12737901 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59103347 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8527819239936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251278455331840 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251278331450368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚242750617479680 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72223368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59103349 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1110862026788848 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚6602.195933217 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1117401630459392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1125881180049920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚242750636091904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845134286200407e+09×¢‘”Ö2·- _wandbruntime‚6602  _runtime‚6602É`ÁÅ'¸üM _runtime‚6602.813316046 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1119477323692751 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _step‚600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8480659689472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30201389 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12739638 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1117549904961536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131344164 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8481001120768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8468206797817 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131354323 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚242783797749760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8480659689472 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1117549904961536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚251312755842048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8528830023680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72233024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8528939480064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1111009457456320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1111009474495680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30201101 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131344164 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59111140 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131354323 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72243181 trainer/global_step‚7299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59111142 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1117549922000896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12739635 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1117549922000896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17461751 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72243181 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528Bȧ½¹N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17461751 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72243181 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _runtime‚6602.813316046 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1119477323692751 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _step‚600 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8480659689472 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30201389 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12739638 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1117549904961536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131344164 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8481001120768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8468206797817 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131354323 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚242783797749760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8480659689472 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1117549904961536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚251312755842048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8528830023680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72233024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8528939480064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1111009457456320 RBDeviceStatsMonitor.on_train_batc%Lð h_end/reserved_bytes.all.allocated‚ 24566038528 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1111009474495680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30201101 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131344164 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59111140 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131354323 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72243181 trainer/global_step‚7299 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59111142 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1117549922000896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12739635 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1117549922000896Ø eب2º- _wandbruntime‚6602  _runtime‚66027¸U¹»‚ ,loss/FBt2/detection_loss‚1.0814005285647 epoch‚18 0loss/FRt2/assignment_loss‚0.7359528486014013 .loss/FRt2/detection_loss‚1.580056371596602 _step‚601 2loss/SRqqt2/assignment_loss‚0.5100444636963716 3loss/FBt1/assignment_loss‚0.0023274419546698046 2loss/FBt2/assignment_loss‚0.039008466481062376 1loss/SRqqt1/detection_loss‚0.6735970025574624 0loss/FBt1/detection_loss‚0.35078589278273287 %loss/total_loss‚8.385086817486448 % _timestamp‚1.7845134292390049e+09 2loss/SRqqt1/assignment_loss‚0.5477228527488845 trainer/global_step‚7299 _runtime‚6602.814450618 0loss/FRt1/assignment_loss‚1.6099808491540677 /loss/FRt1/detection_loss‚0.5060530119527584 1loss/SRqqt2/detection_loss‚0.7481570873957346•XÆ3 ¼‡ 1loss/SRqqt1/detection_loss‚0.6735970025574624 0loss/FBt1/detection_loss‚0.35078589278273287 %loss/total_loss‚8.385086817486448 % _timestamp‚1.7845134292390049e+09 2loss/SRqqt1/assignment_loss‚0.5477228527488845 trainer/global_step‚7299 _runtime‚6602.814450618 0loss/FRt1/assignment_loss‚1.6099808491540677 /loss/FRt1/detection_loss‚0.5060530119527584 1loss/SRqqt2/detection_loss‚0.7481570873957346 ,loss/FBt2/detection_loss‚1.0814005285647 epoch‚18 0loss/FRt2/assignment_loss‚0.7359528486014013 .loss/FRt2/detection_loss‚1.580056371596602 _step‚601 2loss/SRqqt2/assignment_loss‚0.5100444636963716 3loss/FBt1/assignment_loss‚0.0023274419546698046 2loss/FBt2/assignment_loss‚0.039008466481062376ÙñÅOŸ½‚(:” ¡‡öÒåŸÍË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚125.483' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.827666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LL Y¾‚(:Î ¡‡öÒáÚÓÌ disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚39342.84912109375( disk.nvme1c1n1.out‚22551.16064453125 disk.nvme1n1.in‚0 disk.md0.in‚0$ memory_percent‚96.99175357818604( proc.memory.availableMB‚ 492.87109375) disk./.usagePercent‚75.97359845351366 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625 proc.memory.rssMB‚3013.5( proc.memory.percent‚18.3929443359375 cpu‚25.68200447287232& disk.nvme1n1.out‚22551.16064453125 network.sent‚ 1.1603303e+07% disk./.usageGB‚1166.3846702575684& disk.nvme0n1.out‚39342.84912109375" disk.dm-0.out‚50754.7900390625 disk.dm-1.in‚0 network.recv‚ 1.766758e+06 disk.md0.out‚ 198.703125 disk.dm-1.out‚ 10714.40625 proc.cpu.threads‚17”Õ[¿‚(:Ð °‡öÒÛƒÜË& proc.memory.availableMB‚ 492.453125 proc.cpu.threads‚17 network.sent‚ 1.1666498e+07 disk.md0.out‚ 198.9453125 disk.dm-0.in‚ 0.00390625 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚22551.6259765625$ memory_percent‚96.99430465698242 proc.memory.rssMB‚3013.5( disk.nvme0c0n1.out‚39343.53759765625 network.recv‚ 1.771765e+06 disk.md0.in‚0# disk.dm-1.out‚10714.48779296875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚39343.53759765625( proc.memory.percent‚18.3929443359375( disk./.usagePercent‚75.9735972111436" disk.dm-0.out‚50755.3369140625 disk.nvme1n1.in‚0% disk.nvme1n1.out‚22551.6259765625 cpu‚25.681526036189833$ disk./.usageGB‚1166.384651184082 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625"/“±“À‚(:ˆ °‡öÒÿ´™Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.665' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hØOWÁ‚(:Ì ¿‡öÒ£¯¹Ë disk.md0.in‚0) disk./.usagePercent‚75.97360093825377 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1n1.out‚22566.9814453125 disk.md0.out‚ 199.28515625( proc.memory.percent‚18.3929443359375 cpu‚25.68100687204848 network.recv‚ 1.775469e+06$ disk./.usageGB‚1166.384708404541 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚50770.93017578125& disk.nvme0c0n1.out‚39356.603515625 disk.nvme0n1.in‚ 0.00390625$ disk.nvme0n1.out‚39356.603515625 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0# disk.dm-1.out‚10726.56591796875 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚22566.9814453125$ memory_percent‚96.99440002441406$ proc.memory.availableMB‚492.4375 proc.memory.rssMB‚3013.5 proc.cpu.threads‚17 network.sent‚ 1.1669563e+07 disk.nvme1n1.in‚0³ê“‚(:ˆ ¿‡öÒÔ€ŽÌ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚271.452' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚90.484 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SÌò2Ã- _wandbruntime‚6645  _runtime‚6645ÑçÙòÉÄà ' lr-AdamW/pg2‚0.00044281157827377206 trainer/global_step‚7349 % _timestamp‚1.7845134718703783e+09 _runtime‚6645.445980262 _step‚602 ' lr-AdamW/pg1‚0.00044281157827377206FˆÝ-ÎÅÈ _runtime‚6645.445980262 _step‚602 ' lr-AdamW/pg1‚0.00044281157827377206 ' lr-AdamW/pg2‚0.00044281157827377206 trainer/global_step‚7349 % _timestamp‚1.7845134718703783e+09ÚÏU¤-2Æ- _wandbruntime‚6645  _runtime‚6645fÐX›ø'ÇòO ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1124815355566592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1133350474258944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8535118692352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚ 6645.45005571 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1133350474258944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59492999 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7845134718713274e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244409649595904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59492997 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72718163 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132201017 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1126755801224035 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _step‚603 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚7349 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1124815372605952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252993211011072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59492997 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72708020 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8535460116480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72708020 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12824604 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8583580027392 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1133350832722432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244409630983680NJ1ý'È÷O WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1133350474258944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8535118692352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚ 6645.45005571 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1133350474258944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59492999 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7845134718713274e+09 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244409649595904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59492997 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72718163 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132201017 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1126755801224035 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _step‚603 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚7349 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1124815372605952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252993211011072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59492997 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72708020 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8535460116480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72708020 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12824604 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8583580027392 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1133350832722432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244409630983680 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1124815355566592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244442829865984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8584691206144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1126904697880505 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8536570222592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132228625 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12826335 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1118381007869280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1124963647108096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 BìŠI$½9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17576864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚6646.066905225 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1133500217330688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132218466 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1126904340279759 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132228625 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8523690011225 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59500792 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1133500217330688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30403199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244442811253760 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132218466 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1124963647108096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8536228791296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72717676 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8584796468224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1118380990829920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59500790 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1124963630068736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59500792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30403495 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1133499858860032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8536570222592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚253027626334208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚604 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656x(Òó'Ë‚N KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1124963647108096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17576864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚6646.066905225 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1133500217330688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132218466 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1126904340279759 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132228625 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8523690011225 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59500792 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1133500217330688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30403199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244442811253760 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132218466 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1124963647108096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8536228791296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72717676 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8584796468224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1118380990829920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59500790 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1124963630068736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59500792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30403495 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1133499858860032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8536570222592 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚253027626334208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚604 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244442829865984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8584691206144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1126904697880505 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8536570222592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132228625 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12826335 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1118381007869280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ÜÆ|"2Ì- _wandbruntime‚6646  _runtime‚6646)ÿ Í‚ 2loss/FBt2/assignment_loss‚0.021589549349803134 /loss/FBt1/detection_loss‚0.5815930024170093 trainer/global_step‚7349 0loss/FRt2/assignment_loss‚0.8664124025854953 2loss/SRqqt1/assignment_loss‚0.5198612579095202 /loss/FRt2/detection_loss‚1.7015844258450896 epoch‚18 _runtime‚6646.067502496 _step‚605 2loss/FBt1/assignment_loss‚0.008620472440345736 1loss/SRqqt1/detection_loss‚0.6663062454602795 -loss/FBt2/detection_loss‚0.56525365079125 %loss/total_loss‚8.352262123740113 % _timestamp‚1.7845134724922082e+09 2loss/SRqqt2/assignment_loss‚0.5263233239417108 /loss/FRt1/detection_loss‚0.5007475028138625 1loss/SRqqt2/detection_loss‚0.7265833068910027 0loss/FRt1/assignment_loss‚1.6673869832947448 7” · 1loss/SRqqt2/detection_loss‚0.7265833068910027 0loss/FRt1/assignment_loss‚1.6673869832947448 2loss/FBt2/assignment_loss‚0.021589549349803134 /loss/FBt1/detection_loss‚0.5815930024170093 trainer/global_step‚7349 0loss/FRt2/assignment_loss‚0.8664124025854953 2loss/SRqqt1/assignment_loss‚0.5198612579095202 /loss/FRt2/detection_loss‚1.7015844258450896 epoch‚18 _runtime‚6646.067502496 _step‚605 2loss/FBt1/assignment_loss‚0.008620472440345736 1loss/SRqqt1/detection_loss‚0.6663062454602795 -loss/FBt2/detection_loss‚0.56525365079125 %loss/total_loss‚8.352262123740113 % _timestamp‚1.7845134724922082e+09 2loss/SRqqt2/assignment_loss‚0.5263233239417108 /loss/FRt1/detection_loss‚0.5007475028138625Ý­Q¾½\Ï‚(:Ñ ·öÒǦ“Ë% disk./.usageGB‚1166.2563705444336 disk.dm-0.in‚ 0.00390625! disk.dm-1.out‚10726.724609375 disk.nvme0n1.in‚ 0.00390625& proc.memory.availableMB‚ 492.359375 proc.memory.rssMB‚3013.5( proc.memory.percent‚18.3929443359375 network.sent‚ 1.1675435e+07 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚22568.04931640625 disk.md0.in‚0$ memory_percent‚96.99487686157227 cpu‚25.680565137283924 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚39378.2861328125 disk.nvme1n1.in‚0& disk.nvme1n1.out‚22568.04931640625 disk.md0.out‚ 200.00390625) disk./.usagePercent‚75.96524152707596# disk.dm-0.out‚50792.01611328125' disk.nvme0c0n1.out‚39378.2861328125 proc.cpu.threads‚17 network.recv‚ 1.77869e+06ƒF.hžÐ‚(:“ ·öÒƒØÏË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚273.273' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.09100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êzðVÑ‚(:Ë ݇öÒÔ˧Ë! disk.dm-1.out‚10726.822265625 disk.rbd0.out‚0& disk.nvme0c0n1.out‚39475.806640625 disk.nvme0n1.in‚ 0.00390625 cpu‚25.679972261904823 network.recv‚ 1.784101e+06) disk./.usagePercent‚75.99748450873865 disk.nvme1n1.in‚0% disk.nvme1n1.out‚22568.8095703125 disk.md0.out‚ 200.44921875 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme0n1.out‚39475.806640625 disk.nvme1c1n1.in‚0% disk./.usageGB‚1166.7513809204102 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚50889.2412109375# memory_percent‚96.9949722290039 proc.memory.rssMB‚3013.5( proc.memory.percent‚18.3929443359375 proc.cpu.threads‚17 disk.md0.in‚0 disk.rbd0.in‚0' disk.nvme1c1n1.out‚22568.8095703125% proc.memory.availableMB‚ 492.34375 network.sent‚ 1.1738997e+07 disk.dm-1.in‚0š5dˆžÒ‚(:“ ݇öÒÆìéË gpu.0.gpu‚92.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.557' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.18566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ(î92Ó- _wandbruntime‚6688  _runtime‚6688\~R°ÆÔÀ & lr-AdamW/pg2‚0.0004405975630860548 trainer/global_step‚7399 $ _timestamp‚1.784513515374515e+09 _runtime‚6688.950668893 _step‚606 & lr-AdamW/pg1‚0.0004405975630860548~…9ËÕÅ $ _timestamp‚1.784513515374515e+09 _runtime‚6688.950668893 _step‚606 & lr-AdamW/pg1‚0.0004405975630860548 & lr-AdamW/pg2‚0.0004405975630860548 trainer/global_step‚7399Þc±Eæ2Ö- _wandbruntime‚6688  _runtime‚6688í3þù'×óO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1125605093536048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1140820126931456 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133075319 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1132229097713152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133085464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1140819768467968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8590687794176 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246068644487680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59882649 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8639376391168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12911301 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59882647 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254708155246080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8577724274995 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30600874 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59882647 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246068663099904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1140819768467968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8591029218304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59882649 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30600570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 85910292183040DÈb¯ØøO ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1132229080673792 trainer/global_step‚7399 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17689570 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8578064836269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73202815 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 115755008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1134183175411677 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1140820126931456 F8DeviceStatsMonitor.on_train_batch_slwÒO"tart/active.all.freed‚ 133075319 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1132229097713152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133085464 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1140819768467968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8590687794176 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚246068644487680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59882649 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8639376391168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12911301 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59882647 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254708155246080 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8577724274995 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30600874 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59882647 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚246068663099904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1140819768467968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8591029218304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59882649 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30600570 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8591029218304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1125605093536048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17691551 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73202328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254742447958016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59890440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133102927 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1140969511539712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133092768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1132377355175936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚7399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246101843369984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73212485 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8578832663247 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73202328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30604586 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1132377372215296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8640604588032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184²Ý”¡'Û‚N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8592139324416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8591797893120 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134360064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254742313597952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133092768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59890440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30604875 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12913035 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1134331356866767 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1140969511539712 % _timestamp‚1.7845135159989614e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1125752524203520 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1132377372215296 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1140969511539712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133092768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1132377355175936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚7399 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246101843369984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73212485 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8578832663247 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73202328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30604586 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1132377372215296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8640604588032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1132377355175936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17691551 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73202328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254742447958016 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59890440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133102927 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8646598320128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12998001 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8633207488403 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60272297 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1132976643949008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73677324 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚6733.130041108 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1148289421140480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12998004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8695319823872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60272299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 136464384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1148289062676992 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1139642805780992 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1139642805780992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247727676603904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73687467 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30801987 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60272299 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1148289421140480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488ñ± žþ'êøO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73677324 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚6733.130041108 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1148289421140480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12998004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8695319823872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60272299 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 136464384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1148289062676992 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1139642805780992 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1139642805780992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247727676603904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73687467 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30801987 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60272299 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1148289421140480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8646598320128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8695201971712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1139642822820352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60272297 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17804259 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845135595510197e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1139642822820352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚247727657991680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1132976626909648 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8633548049677 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133959766 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8646256896000 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8646598320128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12998001 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8633207488403 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60272297 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1132976643949008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452ã~9êä2ë- _wandbruntime‚6733  _runtime‚6733óxÕ^'ìüM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8647708426240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73686980 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚612 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133967070 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17806556 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8634656438041 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30806004 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256457159539200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30806294 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133977229 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133977229 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8696321277440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8647366994944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60280090 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1139791080283136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73697137 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚247760856873984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8696426539520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1148438447278080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8647366994944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1141758373453775 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1139791097322496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60280090 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12999738 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1148438805748736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60280092 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247760838261760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386[~á׆íN _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1133124074616480 F=DeviceStatsMonitor.on_train÷Š&_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚7449 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1148438805748736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256457283413504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73686980 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133977229 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8696321277440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8647366994944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60280090 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1139791080283136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73697137 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚247760856873984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8696426539520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1148438447278080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8647366994944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1141758373453775 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1139791097322496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60280090 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12999738 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1148438805748736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60280092 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247760838261760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8647708426240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73686980 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚612 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133967070 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17806556 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8634656438041 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30806004 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256457159539200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30806294 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133977229 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1155758356886016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8702167421952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1140348177322608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _step‚615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8751039240192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8701825997824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74172119 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1155758715349504 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249386690107904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8688690701811 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13084701 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8751142412288 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17918914 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1147056547927552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134834068 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74161976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74161976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1147056547927552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31003618 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8689031263085 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚7499 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8702167421952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60661947 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258137832520192 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1149036850985059 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8701825997824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60661949 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚249386671495680 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1155758715349504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74172119 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚ 6777.35204251 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134823923 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17918627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1149037208585693 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896{ü &ôüõO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1155758356886016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31003328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13084704 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚258137710735872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60661949 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1147056530888192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 # _timestamp‚1.78451360377634e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1147056530888192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1155758356886016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8702167421952 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1140348177322608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _step‚615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8751039240192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8701825997824 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74172119 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1155758715349504 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚249386690107904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8688690701811 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13084701 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8751142412288 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17918914 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1147056547927552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134834068 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74161976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74161976 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1147056547927552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31003618 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8689031263085 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚_272 trainer/global_step‚7499 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8702167421952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60661947 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258137832520192 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1149036850985059 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8701825997824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60661949 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚249386671495680 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1155758715349504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74172119 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚ 6777.35204251 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134823923 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17918627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1149037208585693 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303çårÖ2ý- _wandbruntime‚6777  _runtime‚6777µUu6'þüM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17921246 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1155908099957760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1147204805390336 _step‚616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134841372 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74171632 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134841372 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8703277528064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74181789 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249419870377984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1140495607990080 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1147204822429696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134851531 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1149185390040783 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31007406 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8702936096768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60669742 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8752259203584 % _timestamp‚1.7845136043982444e+09 _runtime‚6777.974016164 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8702936096768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60669740 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31007684 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13086438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74181789 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8689799090063 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8703277528064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374iCõ('ÿN NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60669742 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1147204805390336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17920971 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1140495590950720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1149185747641529 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74171632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1155908099957760 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31007406 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8702936096768 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60669742 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8752259203584 % _timestamp‚1.7845136043982444e+09 _runtime‚6777.974016164 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8702936096768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60669740 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31007684 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13086438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74181789 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8689799090063 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8703277528064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17921246 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1155908099957760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1147204805390336 _step‚616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134841372 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74171632 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134841372 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8703277528064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74181789 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚249419870377984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1140495607990080 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1147204822429696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134851531 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1149185390040783 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251045684999680 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1154470255995392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18033925 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1163227651095040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74656771 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31205033 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18033632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74656771 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8744173915219 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251045703611904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31205329 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135698225 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1154470255995392 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61051599 NFDeviceßêÓÍäStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1147719693656848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8744514476493 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8757395099648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚259852699773440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8757736523776 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1156463867572067 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13171404 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8757736523776 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1163228009558528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74646628 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8806890892288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1156464225172701 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1147719710696208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8806996161536 $ _timestamp‚1.784513647306417e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1163227651095040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61051597 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135708370 trainer/global_step‚7549 _step‚619 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74646628 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8757395099648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚259852575891968— Ú§ý'Ž÷O WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6820.881946339 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1156463867572067 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13171404 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8757736523776 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1163228009558528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74646628 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8806890892288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1156464225172701 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1147719710696208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8806996161536 $ _timestamp‚1.784513647306417e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1163227651095040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61051597 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135708370 trainer/global_step‚7549 _step‚619 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74646628 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8757395099648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚259852575891968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1163228009558528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251045684999680 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1154470255995392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18033925 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1163227651095040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74656771 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31205033 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18033632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74656771 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8744173915219 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251045703611904 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31205329 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135698225 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1154470255995392 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61051599 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1147719693656848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8744514476493 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8757395099648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚259852699773440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8757736523776 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135715674 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18036166 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845136479283433e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1154618547536896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74666441 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74656284 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61059392 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259887001406464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1163377394166784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31209012 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259886867046400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8758846629888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8745622864857 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61059390 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135725833 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1154618530497536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8758846629888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2îr0Ç‘‚N RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845136479283433e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1154618547536896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74666441 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74656284 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61059392 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259887001406464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1163377394166784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31209012 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259886867046400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8758846629888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8745622864857 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61059390 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135725833 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1154618530497536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8758846629888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74656284 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1147867124324320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8808117524480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1154618530497536 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135715674 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18036166ìä­„2’- _wandbruntime‚6821  _runtime‚6821rxkÇ“‚ 2loss/FBt2/assignment_loss‚0.002445980272647033 1loss/SRqqt1/detection_loss‚0.7146971577508375 /loss/FBt2/detection_loss‚0.5318057353918751 0loss/FRt1/detection_loss‚0.47163324358519054 %loss/total_loss‚7.738346867219767 /loss/FRt2/assignment_loss‚0.745535735792957 2loss/SRqqt1/assignment_loss‚0.5063437401606934 .loss/FRt2/detection_loss‚1.512053508665675 1loss/SRqqt2/detection_loss‚0.6492712335678434 /loss/FBt1/detection_loss‚0.4619755379086496 epoch‚18 trainer/global_step‚7549 _step‚621 /loss/FRt1/assignment_loss‚1.635324120694983 2loss/SRqqt2/assignment_loss‚0.4981411233035177 2loss/FBt1/assignment_loss‚0.009119750124897635 % _timestamp‚1.7845136479293962e+09 _runtime‚6821.504704737»’e ”‡ 2loss/FBt2/assignment_loss‚0.002445980272647033 1loss/SRqqt1/detection_loss‚0.7146971577508375 /loss/FBt2/detection_loss‚0.5318057353918751 0loss/FRt1/detection_loss‚0.47163324358519054 %loss/total_loss‚7.738346867219767 /loss/FRt2/assignment_loss‚0.745535735792957 2loss/SRqqt1/assignment_loss‚0.5063437401606934 .loss/FRt2/detection_loss‚1.512053508665675 1loss/SRqqt2/detection_loss‚0.6492712335678434 /loss/FBt1/detection_loss‚0.4619755379086496 epoch‚18 trainer/global_step‚7549 _step‚621 /loss/FRt1/assignment_loss‚1.635324120694983 2loss/SRqqt2/assignment_loss‚0.4981411233035177 2loss/FBt1/assignment_loss‚0.009119750124897635 % _timestamp‚1.7845136479293962e+09 _runtime‚6821.504704737íàÄ,Ö^•‚(:Ó óˆöÒÚžÀË disk.rbd0.out‚0% disk.nvme0n1.out‚40214.8935546875 proc.memory.rssMB‚3013.5) disk./.usagePercent‚76.00881467520114" disk.dm-1.out‚10728.0751953125' proc.memory.availableMB‚ 491.4921875( proc.memory.percent‚18.3929443359375 disk.nvme1n1.in‚0 disk.md0.out‚ 204.6640625 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625$ memory_percent‚97.00016975402832 proc.cpu.threads‚17 network.recv‚ 1.823856e+06 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚22821.17529296875 disk.rbd0.in‚0' disk.nvme0c0n1.out‚40214.8935546875 cpu‚25.678875072029765 network.sent‚ 1.1958258e+07% disk./.usageGB‚1166.9253273010254( disk.nvme1c1n1.out‚22821.17529296875 disk.md0.in‚0# disk.dm-0.out‚51870.41064453125˜Õåž–‚(:“ óˆöÒýä°Ì gpu.0.gpu‚97.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.072' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.35733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€¿êM[—‚(:Ð ‚‰öÒ›€¾Ë disk.nvme0n1.in‚ 0.00390625$ disk.nvme0n1.out‚40338.365234375( proc.memory.percent‚18.3929443359375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme1c1n1.out‚22886.43896484375& disk.nvme1n1.out‚22886.43896484375 disk.md0.in‚0 disk.md0.out‚ 204.90234375( proc.memory.availableMB‚ 491.41015625# disk.dm-0.out‚52058.43798828125 disk.dm-1.in‚0 disk.dm-1.out‚10728.25390625$ memory_percent‚97.00067043304443 network.sent‚ 1.2021702e+07 network.recv‚ 1.828395e+06 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0c0n1.out‚40338.365234375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 proc.memory.rssMB‚3013.5 cpu‚25.678447460785367( disk./.usagePercent‚76.0046259003108% disk./.usageGB‚1166.8610191345215 disk.dm-0.in‚ 0.00390625Ô0Hc“˜‚(:ˆ ‚‰öÒÛÉŠÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.972' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.324 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 üõ¯\™‚(:Ñ ‘‰öÒö÷Ë( proc.memory.availableMB‚ 491.41796875 proc.cpu.threads‚17 network.recv‚ 1.832225e+06) disk./.usagePercent‚76.00462515488876 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.md0.out‚ 205.34765625( proc.memory.percent‚18.3929443359375 network.sent‚ 1.2024666e+07% disk./.usageGB‚1166.8610076904297 disk.rbd0.in‚0( disk.nvme0c0n1.out‚40343.99951171875$ memory_percent‚97.00062274932861 cpu‚25.678058255688878 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.dm-1.out‚10728.31640625& disk.nvme1n1.out‚22887.80029296875 proc.memory.rssMB‚3013.5 disk.dm-0.out‚ 52064.4140625& disk.nvme0n1.out‚40343.99951171875 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚22887.80029296875 disk.md0.in‚0í ï’žš‚(:“ ‘‰öÒ××’Ì gpu.0.gpu‚89.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚121.187' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.39566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ W‡2›- _wandbruntime‚6864  _runtime‚6864qbÚÇœÁ & lr-AdamW/pg2‚0.0004316572825074019 trainer/global_step‚7599 % _timestamp‚1.7845136909350162e+09 _runtime‚6864.510587512 _step‚622 & lr-AdamW/pg1‚0.0004316572825074019f†ÏOÌÆ % _timestamp‚1.7845136909350162e+09 _runtime‚6864.510587512 _step‚622 & lr-AdamW/pg1‚0.0004316572825074019 & lr-AdamW/pg2‚0.0004316572825074019 trainer/global_step‚7599îZ“üX2ž- _wandbruntime‚6864  _runtime‚6864ˆI#ù'ŸóO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31406599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136572527 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75141423 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚261567382590976 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8812964201472 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75141423 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚252704698503680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1163890884159075 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _runtime‚6864.513626972 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1155091227030448 _step‚623 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31406311 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252704717115904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8862684087296 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8862791453696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61441249 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75131280 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136582672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1170697303767552 trainer/global_step‚7599 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61441247 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1161883981102592 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1170696945304064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13258104 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8812964201472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13258101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚261567508569600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1161883998141952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75131280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18148495 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1170696945304064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8813305625600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18148210 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1161883981102592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61441247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1155091244069808 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845136909359949e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8813305625600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1161883998141952 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1170696945304064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8813305625600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18148210 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1161883981102592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61441247 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1155091244069808 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845136909359949e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8813305625600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1161883998141952 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75141423 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚252704698503680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1163890884159075 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _runtime‚6864.513626972 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1155091227030448 _step‚623 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31406311 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252704717115904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8862684087296 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8862791453696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61441249 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75131280 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136582672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1170697303767552 trainer/global_step‚7599 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61441247 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1161883981102592 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1170696945304064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13258104 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8812964201472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13258101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚261567508569600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1161883998141952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75131280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18148495 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1164039423214799 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13259838 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13259835 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚261601673753600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252737878773760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1162032272644096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8814415731712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136589976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61449042 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75140936 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75151093 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75151093 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1164039780815545 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1162032255604736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8863794979840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8800765516879 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1162032255604736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61449040 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75140936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚261601803919360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252737897385984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚7599 % _timestamp‚1.7845136915574064e+09 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8863906533376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚6865.133232022 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136600135 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18150782 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136600135 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8814074300416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1170846329905152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31410620 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8814415731712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18150494 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1170846688375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61449042 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1170846688375808 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1162032272644096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31410329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8801106078265òp´¦'£‚N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8863906533376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚6865.133232022 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136600135 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18150782 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136600135 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8814074300416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1170846329905152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31410620 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8814415731712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18150494 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1170846688375808 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61449042 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1170846688375808 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1162032272644096 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31410329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8801106078265 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1164039423214799 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13259838 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13259835 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚261601673753600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252737878773760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1162032272644096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8814415731712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136589976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61449042 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75140936 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75151093 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75151093 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1164039780815545 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1162032255604736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8863794979840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8800765516879 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1162032255604736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61449040 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75140936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚261601803919360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252737897385984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚7599 % _timestamp‚1.7845136915574064e+09 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386ð5Ï \2¤- _wandbruntime‚6865  _runtime‚6865³?¥ˆ 3loss/SRqqt1/assignment_loss‚0.49361079060301605 /loss/FRt1/detection_loss‚0.4673811185588706 1loss/SRqqt2/detection_loss‚0.6302445598108612 0loss/FBt1/detection_loss‚0.42972043600486126 %loss/total_loss‚7.813948112300075 % _timestamp‚1.7845136915585227e+09 _step‚625 2loss/SRqqt2/assignment_loss‚0.4945349663250852 3loss/FBt1/assignment_loss‚0.0074873951204031935 3loss/FBt2/assignment_loss‚0.0015225201397841362 epoch‚18 _runtime‚6865.137020744 0loss/FRt2/assignment_loss‚0.7648433118331723 /loss/FRt2/detection_loss‚1.5645614240613834 0loss/FRt1/assignment_loss‚1.7158798977718372 1loss/SRqqt1/detection_loss‚0.7489746862976131 /loss/FBt2/detection_loss‚0.4951870057731867 trainer/global_step‚7599Wi$§¦ %loss/total_loss‚7.813948112300075 % _timestamp‚1.7845136915585227e+09 _step‚625 2loss/SRqqt2/assignment_loss‚0.4945349663250852 3loss/FBt1/assignment_loss‚0.0074873951204031935 3loss/FBt2/assignment_loss‚0.0015225201397841362 epoch‚18 _runtime‚6865.137020744 0loss/FRt2/assignment_loss‚0.7648433118331723 /loss/FRt2/detection_loss‚1.5645614240613834 0loss/FRt1/assignment_loss‚1.7158798977718372 1loss/SRqqt1/detection_loss‚0.7489746862976131 /loss/FBt2/detection_loss‚0.4951870057731867 trainer/global_step‚7599 3loss/SRqqt1/assignment_loss‚0.49361079060301605 /loss/FRt1/detection_loss‚0.4673811185588706 1loss/SRqqt2/detection_loss‚0.6302445598108612 0loss/FBt1/detection_loss‚0.42972043600486126ñâJ\§‚(:Ñ  ‰öÒ›¹æË) disk./.usagePercent‚76.00462267014865$ disk.nvme1c1n1.out‚ 22888.2109375 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0# disk.dm-1.out‚10728.39306640625( disk.nvme0c0n1.out‚40344.63037109375 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚40344.63037109375 network.sent‚ 1.2043379e+07 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚97.01335430145264( proc.memory.availableMB‚ 489.33203125) proc.memory.percent‚18.41108798980713 cpu‚25.67937382919457 network.recv‚ 1.835626e+06" disk.nvme1n1.out‚ 22888.2109375 disk.rbd0.in‚0# proc.memory.rssMB‚ 3016.47265625 proc.cpu.threads‚17$ disk./.usageGB‚1166.860969543457 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.md0.out‚ 205.55859375 disk.dm-0.out‚ 52064.921875ÄlÅuŸ¨‚(:”  ‰öÒ€‘§Ì gpu.0.gpu‚97.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.004' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.001333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É5‹&2©- _wandbruntime‚6881  _runtime‚6881ð[†,,ªôz 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6442038416862488 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16208267211914062 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8168368339538574 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04475494101643562 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1909809112548828 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8966553807258606 *jet/accuracy_2_of_3‚0.5819031596183777 6validation_average_jet_accuracy‚0.3699716627597809 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21208074688911438 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8981481194496155 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7120782136917114 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.844967246055603 trainer/global_step‚7599 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5017971992492676 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21208074688911438 +particle/sensitivity‚0.6440908908843994 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45633530616760254 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38516268134117126 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7456362843513489 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6440070867538452 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7223617434501648 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_5_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967668652534485 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7359658479690552 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9115391969680786 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15724167227745056 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6412369608879089 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19094260036945343 jet/accuracy_4_of_5‚0 _runtime‚6881.283181996 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.844967246055603 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚626 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2794956862926483 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5843288898468018 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7012128233909607 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.996094286441803 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.27761712670326233 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22546923160552979 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2557271718978882 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900867938995361 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7340124845504761 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_1_of_1‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6887621879577637 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3806794583797455 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6802237033843994 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22582709789276123 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8809812664985657 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20485003292560577 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972329139709473 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06734885275363922 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 epoch‚18 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6828150749206543 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4947916865348816 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23199042677879333 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6440070867538452 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8981481194496155 particle/accuracy_1_of_3‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8955782651901245 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 -particle/accuracy_4_of_4‚0.95782071352005 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9083144664764404 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17198403179645538 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20945483446121216 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 (particle/accuracy‚0.8559918403625488 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6942118406295776 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963704943656921 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3092832863330841 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21376964449882507 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04975244030356407 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 *jet/accuracy_1_of_4‚0.9820159077644348 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6928289532661438 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22950683534145355 particle/accuracy_1_of_5‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *particle/specificity‚0.913946807384491 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5174123048782349 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03517020493745804 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7868356108665466 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7354157567024231 +jet/accuracy_3_of_4‚0.09795041382312775 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 particle/accuracy_2_of_4‚1 jet/accuracy_5_of_5‚0 $ _timestamp‚1.784513707706861e+09 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.516793429851532 )jet/accuracy_2_of_4‚0.789550244808197 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20422948896884918 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7984780669212341 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3181663155555725 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4865081012248993 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7739125490188599 particle/accuracy_3_of_3‚1 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.630555272102356 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7317708134651184 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1633484661579132 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03713223710656166 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03079489804804325 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20488929748535156 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2215341329574585 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.043349944055080414 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6704375743865967 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7888360023498535 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9299118518829346 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7518248558044434 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_2‚0.1644335687160492 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5799813270568848 '·Ÿ–­Nparticle/f_score‚0.6576348543167114 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 *loss/val_total_loss‚7.7451476783437485 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7613449096679688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02353314496576786 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968811869621277 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22841830551624298 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6540642976760864 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17411302030086517 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2058333456516266 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6874353885650635 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.049265842884778976 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6556071043014526 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6504108309745789 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902462959289551 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20570160448551178 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9592127799987793 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4624776840209961 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.043070435523986816 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9015306830406189 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5666666626930237 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7415105700492859 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8961741328239441 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14891760051250458 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03079489804804325 +jet/accuracy_3_of_3‚0.07835932821035385 jet/accuracy_2_of_5‚1 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.766496479511261 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7091865539550781 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7687749266624451 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953127503395081 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6241060495376587 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 +jet/accuracy_1_of_1‚0.32215726375579834 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.896636426448822 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6775708198547363 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7330352663993835 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19097523391246796 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7217700481414795 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6782880425453186 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22519448399543762 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04889889061450958 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6778032779693604 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968¦@¥=«ùz 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19094260036945343 jet/accuracy_4_of_5‚0 _runtime‚6881.283181996 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.844967246055603 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚626 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2794956862926483 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5843288898468018 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7012128233909607 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.996094286441803 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.27761712670326233 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22546923160552979 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2557271718978882 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900867938995361 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7340124845504761 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_1_of_1‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6887621879577637 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3806794583797455 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6802237033843994 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22582709789276123 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8809812664985657 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20485003292560577 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972329139709473 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06734885275363922 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 epoch‚18 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6828150749206543 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4947916865348816 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23199042677879333 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6440070867538452 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8981481194496155 particle/accuracy_1_of_3‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8955782651901245 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 -particle/accuracy_4_of_4‚0.95782071352005 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9083144664764404 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17198403179645538 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20945483446121216 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 (particle/accuracy‚0.8559918403625488 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6942118406295776 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963704943656921 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3092832863330841 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21376964449882507 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04975244030356407 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 *jet/accuracy_1_of_4‚0.9820159077644348 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6928289532661438 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22950683534145355 particle/accuracy_1_of_5‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *particle/specificity‚0.913946807384491 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5174123048782349 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03517020493745804 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7868356108665466 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7354157567024231 +jet/accuracy_3_of_4‚0.09795041382312775 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 particle/accuracy_2_of_4‚1 jet/accuracy_5_of_5‚0 $ _timestamp‚1.784513707706861e+09 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.516793429851532 )jet/accuracy_2_of_4‚0.789550244808197 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20422948896884918 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7984780669212341 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3181663155555725 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4865081012248993 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7739125490188599 particle/accuracy_3_of_3‚1 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.630555272102356 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7317708134651184 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1633484661579132 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03713223710656166 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03079489804804325 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20488929748535156 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2215341329574585 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.043349944055080414 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6704375743865967 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7888360023498535 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9299118518829346 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7518248558044434 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_2‚0.1644335687160492 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5799813270568848 'particle/f_score‚0.6576348543167114 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 *loss/val_total_loss‚7.7451476783437485 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7613449096679688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02353314496576786 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968811869621277 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22841830551624298 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6540642976760864 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17411302030086517 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2058333456516266 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6874353885650635 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.049265842884778976 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6556071043014526 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6504108309745789 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902462959289551 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20570160448551178 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9592127799987793 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4624776840209961 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.043070435523986816 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9015306830406189 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5666666626930237 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7415105700492859 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8961741328239441 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14891760051250458 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03079489804804325 +jet/accuracy_3_of_3‚0.07835932821035385 jet/accuracy_2_of_5‚1 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.766496479511261 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7091865539550781 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7687749266624451 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953127503395081 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6241060495376587 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 +jet/accuracy_1_of_1‚0.32215726375579834 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.896636426448822 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6775708198547363 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7330352663993835 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19097523391246796 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7217700481414795 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6782880425453186 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22519448399543762 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04889889061450958 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6778032779693604 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6442038416862488 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16208267211914062 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8168368339538574 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04475494101643562 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1909809112548828 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8966553807258606 *jet/accuracy_2_of_3‚0.5819031596183777 6validation_average_jet_accuracy‚0.3699716627597809 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21208074688911438 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8981481194496155 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7120782136917114 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.844967246055603 trainer/global_step‚7599 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5017971992492676 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21208074688911438 +particle/sensitivity‚0.6440908908843994 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45633530616760254 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38516268134117126 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7456362843513489 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6440070867538452 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7223617434501648 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_5_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967668652534485 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7359658479690552 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9115391969680786 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15724167227745056 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6412369608879089 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437òp„›;^¬‚(:Ó ¯‰öÒäÒúË disk.nvme1n1.in‚0 disk.md0.out‚ 205.9609375 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚40349.0576171875' proc.memory.availableMB‚ 489.3046875# proc.memory.rssMB‚ 3016.47265625 cpu‚25.683474714790187$ disk./.usageGB‚1166.861499786377 disk.nvme1c1n1.in‚0$ disk.nvme1c1n1.out‚ 22889.7109375 disk.md0.in‚0# disk.dm-0.out‚52069.90380859375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚97.01352119445801 network.sent‚ 1.2104936e+07 network.recv‚ 1.841071e+06) disk./.usagePercent‚76.00465720803628 disk.dm-0.in‚ 0.00390625) proc.memory.percent‚18.41108798980713 proc.cpu.threads‚17" disk.nvme1n1.out‚ 22889.7109375# disk.dm-1.out‚10728.48681640625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚40349.0576171875P· Až­‚(:“ ¯‰öÒ‘é‡Í gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.511' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.17033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ¯‰Ûf®‚(:Û ¾‰öÒº÷ÅË disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚22890.56689453125) proc.memory.percent‚18.41108798980713 network.sent‚ 1.2151161e+07% disk./.usageGB‚1166.8613891601562# disk.dm-0.out‚52070.55224609375" disk.dm-1.out‚10728.7080078125' disk.nvme0c0n1.out‚40350.0830078125 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚40350.0830078125( proc.memory.availableMB‚ 489.30078125 disk.rbd0.in‚0 disk.md0.out‚ 206.43359375# proc.memory.rssMB‚ 3016.47265625 cpu‚25.68300767576506 proc.cpu.threads‚17 network.recv‚ 1.84647e+06) disk./.usagePercent‚76.00465000228994 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚22890.56689453125 disk.nvme1n1.in‚0 disk.md0.in‚0$ memory_percent‚97.01354503631592®òÝ¡¯‚(:’ ¾‰öÒ””ÿË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.473' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.824333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š„h°‚(:Ý ͉öÒ€“ÔË network.recv‚ 1.850114e+06 disk.md0.out‚ 207.05859375" disk.dm-1.out‚10728.7626953125& disk.nvme0n1.out‚40352.44580078125 disk.nvme1n1.in‚0& disk.nvme1n1.out‚22891.48095703125$ memory_percent‚97.01352119445801) proc.memory.percent‚18.41108798980713 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 3016.47265625 proc.cpu.threads‚17 network.sent‚ 1.2156018e+07) disk./.usagePercent‚76.00465323245209% disk./.usageGB‚1166.8614387512207" disk.dm-0.out‚52072.4384765625( disk.nvme0c0n1.out‚40352.44580078125( disk.nvme1c1n1.out‚22891.48095703125 cpu‚25.682574684959317 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625' proc.memory.availableMB‚ 489.3046875ɧ‡Qž±‚(:“ ͉öÒ¿ ‘Ì gpu.0.gpu‚75.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.727' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.57566666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚089™#2²- _wandbruntime‚6924  _runtime‚6924ÐZê¨É³Ã % _timestamp‚1.7845137506480374e+09 _runtime‚6924.225042595 _step‚627 ' lr-AdamW/pg1‚0.00042940187518229707 ' lr-AdamW/pg2‚0.00042940187518229707 trainer/global_step‚7649q§4Î´È ' lr-AdamW/pg1‚0.00042940187518229707 ' lr-AdamW/pg2‚0.00042940187518229707 trainer/global_step‚7649 % _timestamp‚1.7845137506480374e+09 _runtime‚6924.225042595 _step‚627óÃøó92µ- _wandbruntime‚6924  _runtime‚6924F˜þçضóO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31647037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263325586862592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚7649 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1171240050085376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1171240050085376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1173259818203838 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚6924.225109865 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13348917 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚628 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8888851139072 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61953707 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61953705 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8889192563200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263325465078272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8939277580288 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75697426 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75697426 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8875444190932 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1180129259687936 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1171240067124736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8875784752206 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137661276 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8888851139072 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1180129259687936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8889192563200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18298117 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚254386290670080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M@DeviceStatsMoniQº”!tor.on_train_batch_start/inactive_split.all.freed‚31646744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13348920 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137651131 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1180128901224448 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1180128901224448 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1164384016412272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 % _timestamp‚1.7845137506487734e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1173259818203838 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚6924.225109865 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13348917 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚628 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8888851139072 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61953707 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61953705 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8889192563200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263325465078272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8939277580288 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75697426 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75697426 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8875444190932 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1180129259687936 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1171240067124736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8875784752206 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 137661276 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8888851139072 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1180129259687936 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8889192563200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18298117 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚254386290670080 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31646744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13348920 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137651131 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1180128901224448 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1180128901224448 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1164384016412272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 % _timestamp‚1.7845137506487734e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚7649 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1171240050085376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232ôA ¿G2¸- _wandbruntime‚6924  _runtime‚6924-Y$}'¹üM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8889961238016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137678739 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61961500 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75707082 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13350654 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263359879588864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚263359762006016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚254419470940160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1173408357259674 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8940390036480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137668580 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845137512703254e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31651043 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61961498 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75707082 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1171388341626880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137678739 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18300389 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8890302669312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8890302669312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8940291065856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8876893140570 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1171388324587520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8889961238016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61961498 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61961500 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1173407999658928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _step‚629 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1171388324587520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚254419489552384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1180278644296192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31650769 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7649 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18300118 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386ý‘÷Ñ'ºN UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8889961238016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137678739 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61961500 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75707082 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13350654 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263359879588864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚263359762006016 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚254419470940160 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1173408357259674 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8940390036480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137668580 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845137512703254e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31651043 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61961498 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75707082 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1171388341626880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137678739 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18300389 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8890302669312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8890302669312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8940291065856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8876893140570 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1171388324587520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8889961238016 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61961498 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61961500 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1173407999658928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _step‚629 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1171388324587520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚254419489552384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1180278644296192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31650769 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7649 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18300118 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386õ5{& 2»- _wandbruntime‚6924  _runtime‚6924ª2:hC¼‚ /loss/FRt1/assignment_loss‚1.604894692488272 0loss/FRt2/assignment_loss‚0.7247555386142323 2loss/SRqqt1/assignment_loss‚0.5564193134955215 /loss/FRt1/detection_loss‚0.5001610625078373 %loss/total_loss‚7.930889715126374 epoch‚19 trainer/global_step‚7649 % _timestamp‚1.784513751271434|¡¥PÅ5e+09 2loss/FBt1/assignment_loss‚0.024736886712121285 /loss/FRt2/detection_loss‚1.4864224915828494 /loss/FBt1/detection_loss‚0.4587017087577407 _step‚630 2loss/SRqqt2/assignment_loss‚0.5665249991680058 3loss/FBt2/assignment_loss‚0.0008938567253049915 1loss/SRqqt1/detection_loss‚0.6629921722440233 .loss/FBt2/detection_loss‚0.682848101648103 _runtime‚6924.846722235 0loss/SRqqt2/detection_loss‚0.661538891182362ãwŠ# ½‡ 0loss/FRt2/assignment_loss‚0.7247555386142323 2loss/SRqqt1/assignment_loss‚0.5564193134955215 /loss/FRt1/detection_loss‚0.5001610625078373 %loss/total_loss‚7.930889715126374 epoch‚19 trainer/global_step‚7649 % _timestamp‚1.7845137512714345e+09 2loss/FBt1/assignment_loss‚0.024736886712121285 /loss/FRt2/detection_loss‚1.4864224915828494 /loss/FBt1/detection_loss‚0.4587017087577407 _step‚630 2loss/SRqqt2/assignment_loss‚0.5665249991680058 3loss/FBt2/assignment_loss‚0.0008938567253049915 1loss/SRqqt1/detection_loss‚0.6629921722440233 .loss/FBt2/detection_loss‚0.682848101648103 _runtime‚6924.846722235 0loss/SRqqt2/detection_loss‚0.661538891182362 /loss/FRt1/assignment_loss‚1.604894692488272ö|"Ýd¾‚(:Ù ܉öÒ»–ýË# disk./.usageGB‚1164.14404296875 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚40353.10302734375 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0" disk.dm-0.out‚52073.0009765625$ memory_percent‚97.01395034790039& proc.memory.availableMB‚ 489.234375 proc.cpu.threads‚17 disk.md0.out‚ 207.2421875 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0& disk.nvme0n1.out‚40353.10302734375( disk.nvme1c1n1.out‚22892.06201171875# proc.memory.rssMB‚ 3016.47265625) proc.memory.percent‚18.41108798980713) disk./.usagePercent‚75.82765301864517 disk.md0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚10729.0146484375 disk.rbd0.in‚0& disk.nvme1n1.out‚22892.06201171875 cpu‚25.682141959807613 network.sent‚ 1.2171884e+07 network.recv‚ 1.853515e+06ÖØôçž¿‚(:“ ܉öÒ¤¼Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.557' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.18566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃuÂâdÀ‚(:Ù ë‰öÒò¶±Ë$ memory_percent‚97.01395034790039# proc.memory.rssMB‚ 3016.47265625 cpu‚25.681718150587475) disk./.usagePercent‚75.82765326711919% disk./.usageGB‚1164.1440467834473 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚22892.84228515625& disk.nvme1n1.out‚22892.84228515625# disk.dm-0.out‚52076.70947265625 disk.rbd0.out‚0& proc.memory.availableMB‚ 489.234375 network.recv‚ 1.858803e+06 disk.nvme0n1.in‚ 0.00390625$ disk.nvme0n1.out‚40357.177734375 disk.md0.out‚ 207.76171875 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.nvme1n1.in‚0 disk.md0.in‚0" disk.dm-1.out‚10729.0537109375) proc.memory.percent‚18.41108798980713 network.sent‚ 1.2221513e+07 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0c0n1.out‚40357.177734375 disk.dm-1.in‚0¶üÔažÁ‚(:“ ë‰öÒ ºÌ gpu.0.gpu‚63.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.571' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.52366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 wNa‚(:Ö ú‰öÒèß¹Ë disk.nvme1n1.in‚0$ disk.nvme1n1.out‚22893.755859375$ memory_percent‚97.01399803161621) proc.memory.percent‚18.41108798980713# disk.dm-1.out‚10729.18603515625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0' proc.memory.availableMB‚ 489.2265625 cpu‚25.681255248569986 proc.cpu.threads‚17% disk./.usageGB‚1164.1451377868652# disk.dm-0.out‚52077.92041015625$ disk.nvme0n1.out‚40358.443359375# proc.memory.rssMB‚ 3016.47265625 disk.nvme0c0n1.in‚ 0.00390625 network.sent‚ 1.2227411e+07 network.recv‚ 1.862055e+06) disk./.usagePercent‚75.82772433068655 disk.md0.in‚0 disk.md0.out‚ 208.1484375 disk.dm-0.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚40358.443359375& disk.nvme1c1n1.out‚22893.755859375ÆɜÂ(:‘ ú‰öÒÞæÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚278.884' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.96133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îC‹2Ä- _wandbruntime‚6967  _runtime‚6967ªãÄ;ÇÅÁ & lr-AdamW/pg1‚0.0004271386219308022 & lr-AdamW/pg2‚0.0004271386219308022 trainer/global_step‚7699 % _timestamp‚1.7845137938046904e+09 _runtime‚6967.380281221 _step‚631Ë.NðÌÆÆ % _timestamp‚1.7845137938046904e+09 _runtime‚6967.380281221 _step‚631 & lr-AdamW/pg1‚0.0004271386219308022 & lr-AdamW/pg2‚0.0004271386219308022 trainer/global_step‚7699÷šdÚÁ2Ç- _wandbruntime‚6967  _runtime‚69678±[ù'ÈóO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76182078 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256045304174080 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76192221 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18413379 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1187598195433472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚265040381464576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76182078 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8944420240896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62343355 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31848996 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1178653792231936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8944761665024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8995186754048 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1178653775192576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31849282 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62343355 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8944420240896 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1187598553896960 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1187598195433472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62343357 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8995077290496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8930927404340 % _timestamp‚1.7845137938056695e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1180686834790846 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1187598553896960 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138525433 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1180686477190212 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7699 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18413662 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256045322786304 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1171755566825232 _runtime‚6967.382544415 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13435617 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8944761665024 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8931267965614 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1171755549785872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13435620 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76192221 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚265040509540352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62343357 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0…_¶`þ'ÉøO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1180686477190212 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚7699 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18413662 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256045322786304 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1171755566825232 _runtime‚6967.382544415 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13435617 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8944761665024 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8931267965614 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1171755549785872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13435620 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76192221 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚265040509540352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62343357 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76182078 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚256045304174080 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76192221 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18413379 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1187598195433472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚265040381464576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76182078 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8944420240896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62343355 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31848996 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1178653792231936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8944761665024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8995186754048 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1178653775192576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31849282 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62343355 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8944420240896 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1187598553896960 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1187598195433472 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62343357 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8995077290496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8930927404340 % _timestamp‚1.7845137938056695e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1180686834790846 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1187598553896960 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138525433 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452ø¨ý¶S2Ê- _wandbruntime‚6968  _runtime‚6968È€EÊËüM T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1187747580034560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8945871771136 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8932376353978 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8996299212288 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31853292 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8945530339840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256078503056384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62351150 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1178802066734080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1171902980453344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76201891 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62351150 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13437351 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1187747938505216 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1178802049694720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8945530339840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18415652 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76191734 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8945871771136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265074672102912 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717¦e 119488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚7699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8996187658752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138553041 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31853003 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚265074802268672 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62351148 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _step‚633 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784513794426819e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1187747938505216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62351148 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8996299212288 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31853292 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8945530339840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚256078503056384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62351150 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1178802066734080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1171902980453344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76201891 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62351150 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13437351 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1187747938505216 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1178802049694720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8945530339840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18415652 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76191734 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8945871771136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265074672102912 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚7699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8996187658752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138553041 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31853003 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚265074802268672 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62351148 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _step‚633 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784513794426819e+09ùS<ÜI2Í- _wandbruntime‚6968  _runtime‚6968»A4 · 2loss/FBt1/assignment_loss‚0.015839194923923883 0loss/FBt2/detection_loss‚0.46391418898154213 _step‚634 2loss/SRqqt1/assignment_loss‚0.5155662642702034 3loss/FBt2/assignment_loss‚0.0027944760197536746 /loss/FBt1/detection_loss‚0.4678230141042591 %loss/total_loss‚7.908738063177857 0loss/FRt2/assignment_loss‚0.8528507460247386 2loss/SRqqt2/assignment_loss‚0.5493968039221921 0loss/FRt1/detection_loss‚0.47611310761781295 /loss/FRt2/detection_loss‚1.5426070497918152 1loss/SRqqt2/detection_loss‚0.6798060139599116 epoch‚19 0loss/FRt1/assignment_loss‚1.6375799638194006 1loss/SRqqt1/detection_loss‚0.7044472397423039 trainer/global_step‚7699 % _timestamp‚1.7845137944278708e+09 _runtime‚6968.003215053ólO§ÏŒ 3loss/FBt2/assignment_loss‚0.0027944760197536746 /loss/FBt1/detection_loss‚0.4678230141042591 %loss/total_loss‚7.908738063177857 0loss/FRt2/assignment_loss‚0.8528507460247386 2loss/SRqqt2/assignment_loss‚0.5493968039221921 0loss/FRt1/detection_loss‚0.47611310761781295 /loss/FRt2/detection_loss‚1.5426070497918152 1loss/SRqqt2/detection_loss‚0.6798060139599116 epoch‚19 0loss/FRt1/assignment_loss‚1.6375799638194006 1loss/SRqqt1/detection_loss‚0.7044472397423039 trainer/global_step‚7699 % _timestamp‚1.7845137944278708e+09 _runtime‚6968.003215053 2loss/FBt1/assignment_loss‚0.015839194923923883 0loss/FBt2/detection_loss‚0.46391418898154213 _step‚634 2loss/SRqqt1/assignment_loss‚0.5155662642702034úÖH{Ü\Ђ(:Ñ ‰ŠöÒ³üÝË disk.nvme0n1.in‚ 0.00390625% disk.nvme1c1n1.out‚22895.73046875# disk.nvme1n1.out‚22895.73046875 disk.md0.in‚0 disk.md0.out‚ 208.78125' proc.memory.availableMB‚ 489.1953125) proc.memory.percent‚18.41108798980713 network.recv‚ 1.86555e+06 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚40366.42236328125$ memory_percent‚97.01418876647949 proc.cpu.threads‚17) disk./.usagePercent‚75.82772457916056 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0& disk.nvme0n1.out‚40366.42236328125 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 3016.47265625% disk./.usageGB‚1164.1451416015625" disk.dm-0.out‚52086.4599609375 disk.dm-1.in‚0# disk.dm-1.out‚10729.26806640625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 cpu‚25.68075693073084 network.sent‚ 1.22451e+07ßԞт(:“ ‰ŠöÒù‰œÌ gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚280.591' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.53033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßÿÔ9eÒ‚(:Ú ˜ŠöÒŸÝÜË disk.rbd0.in‚0( disk.nvme1c1n1.out‚22896.19384765625( disk.nvme0c0n1.out‚40395.84228515625 network.recv‚ 1.870357e+06) disk./.usagePercent‚75.95847011330733% disk./.usageGB‚1166.1524124145508 disk.md0.in‚0 disk.md0.out‚ 209.01953125 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0! disk.dm-1.out‚10729.349609375 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚40395.84228515625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚22896.19384765625$ memory_percent‚97.01495170593262# proc.memory.rssMB‚ 3016.47265625 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 0.00390625 disk.dm-1.in‚0 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 489.0703125) proc.memory.percent‚18.41108798980713 cpu‚25.680227251308924 network.sent‚ 1.229548e+07! disk.dm-0.out‚52115.724609375¢—s&ŸÓ‚(:” ˜ŠöÒ¡˜¥Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.536' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.178666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%øvõžÔ‚(:“ §ŠöÒý¨ºË gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚268.369' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.45633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z:íâbÕ‚(:× §ŠöÒ‰™Ì# proc.memory.rssMB‚ 3016.47265625 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚54005.21240234375 disk.nvme0n1.in‚ 0.00390625& disk.nvme1n1.out‚22900.19873046875 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚18.41108798980713 network.recv‚ 1.874029e+06) disk./.usagePercent‚76.00763218737913 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0! disk.dm-1.out‚10732.310546875 cpu‚25.680230256388047 network.sent‚ 1.2300333e+07% disk./.usageGB‚1166.9071731567383% disk.nvme0n1.out‚42285.5068359375( disk.nvme1c1n1.out‚22900.19873046875 disk.md0.in‚0 disk.md0.out‚ 209.5859375 disk.nvme0c0n1.in‚ 0.00390625 proc.cpu.threads‚17 disk.dm-1.in‚0' disk.nvme0c0n1.out‚42285.5068359375$ memory_percent‚97.01499938964844$ proc.memory.availableMB‚489.0625uK®2Ö- _wandbruntime‚7011  _runtime‚7011¢-ZÇ×Á & lr-AdamW/pg1‚0.0004248676681174443 & lr-AdamW/pg2‚0.0004248676681174443 trainer/global_step‚7749 % _timestamp‚1.7845138376166549e+09 _runtime‚7011.192285774 _step‚635ÝøÔ#ÌØÆ _runtime‚7011.192285774 _step‚635 & lr-AdamW/pg1‚0.0004248676681174443 & lr-AdamW/pg2‚0.0004248676681174443 trainer/global_step‚7749 % _timestamp‚1.7845138376166549e+09ûç#¤2Ù- _wandbruntime‚7011  _runtime‚7011H­S ø'ÚòO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62733007 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8999989342720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1195067489642496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32051011 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚636 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62733005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8986410617748 trainer/global_step‚7749 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8999989342720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139399735 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1186067517339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266755364212224 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9000330766848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13522320 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1179127100198832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 $ _timestamp‚1.784513837617405e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76676873 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32050707 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1195067848105984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1186067500299776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257704336290304 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1195067489642496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1186067517339136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1186067500299776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76666730 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1179127083159472 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚7011.192893925 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257704317678080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9051027921920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62733005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9050926846976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8986751179022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13522317 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76666730 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139409880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9000330766848 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1188113851377854 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18528691 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266755244525056 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1195067848105984Iü4.ß Û÷O cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9051027921920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62733005 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9050926846976 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8986751179022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13522317 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76666730 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139409880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9000330766848 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1188113851377854 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D;DeviceStatsMonitor.on_train_batch_startZNjN/segment.all.current‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18528691 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266755244525056 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1195067848105984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62733007 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8999989342720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1195067489642496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32051011 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚636 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62733005 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8986410617748 trainer/global_step‚7749 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8999989342720 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139399735 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1186067517339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266755364212224 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9000330766848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13522320 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1179127100198832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 $ _timestamp‚1.784513837617405e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76676873 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32050707 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1195067848105984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1186067500299776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257704336290304 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1195067489642496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1186067517339136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1186067500299776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76666730 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1179127083159472 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚7011.192893925 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257704317678080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ü ~ˆ{2Ü- _wandbruntime‚7011  _runtime‚7011úåV'ÝüM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76686543 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8987859567386 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32054714 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9052041679872 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1179274530866304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚7749 _runtime‚7011.806246865 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266789539628032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266789663502336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76676386 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257737516560384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1186215774801920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32055001 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1179274513826944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139427343 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1195217232714240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1188262032832944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139417184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9001099441664 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1195217232714240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139427343 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62740800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62740798 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1186215791841280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9001440872960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13524054 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8987519006000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 $ _timestamp‚1.784513838229776e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1195216874243584 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1186215774801920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18530947 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13524051 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62740800 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0D&Zõ'ÞN SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1195216874243584 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1186215774801920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18530947 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13524051 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62740800 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76686543 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8987859567386 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32054714 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9052041679872 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1179274530866304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚7749 _runtime‚7011.806246865 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266789539628032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266789663502336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76676386 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257737516560384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1186215774801920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32055001 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1179274513826944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139427343 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1195217232714240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1188262032832944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139417184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9001099441664 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1195217232714240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139427343 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62740800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62740798 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1186215791841280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9001440872960 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13524054 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8987519006000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 $ _timestamp‚1.784513838229776e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ýx 12ß- _wandbruntime‚7011  _runtime‚7011XE`> à‡ % _timestamp‚1.7845138382308974e+09 epoch‚19 3loss/SRqqt1/assignment_loss‚0.47935799916926985 3loss/FBt1/assignment_loss‚0.0007090259769060052 /loss/FRt1/detection_loss‚0.4585806014447041 1loss/SRqqt2/detection_loss‚0.6922386096349649 trainer/global_step‚7749 _runtime‚7011.806840336 2loss/SRqqt2/assignment_loss‚0.5480924572266793 3loss/FBt2/assignment_loss‚6.066269832281196e-05 /loss/FBt1/detection_loss‚0.5088708526158336 0loss/FBt2/detection_loss‚0.48608283650329603 $loss/total_loss‚8.07469225135119 0loss/FRt1/assignment_loss‚1.7885042933273498 0loss/FRt2/assignment_loss‚0.8203753545170739 /loss/FRt2/detection_loss‚1.6743184147481487 1loss/SRqqt1/detection_loss‚0.6175011434886408 _step‚638ž­Êጠ0loss/FRt1/assignment_loss‚1.7885042933273498 0loss/FRt2/assignment_loss‚0.8203753545170739 /loss/FRt2/detection_loss‚1.6743184147481487 1loss/SRqqt1/detection_loss‚0.6175011434886408 _step‚638 % _timestamp‚1.7845138382308974e+09 epoch‚19 3loss/SRqqt1/assignment_loss‚0.47935799916926985 3loss/FBt1/assignment_loss‚0.0007090259769060052 /loss/FRt1/detection_loss‚0.4585806014447041 1loss/SRqqt2/detection_loss‚0.6922386096349649 trainer/global_step‚7749 _runtime‚7011.806840336 2loss/SRqqt2/assignment_loss‚0.5480924572266793 3loss/FBt2/assignment_loss‚6.066269832281196e-05 /loss/FBt1/detection_loss‚0.5088708526158336 0loss/FBt2/detection_loss‚0.48608283650329603 $loss/total_loss‚8.07469225135119þÐ|§]â‚(:Ò ¶ŠöÒ‚ÀõË disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚97.0160961151123 network.sent‚ 1.2316093e+07( disk./.usagePercent‚76.0076185213085 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 209.9921875' proc.memory.availableMB‚ 488.8828125) proc.memory.percent‚18.41108798980713 network.recv‚ 1.877278e+06$ disk.nvme0n1.out‚42286.404296875# disk.dm-1.out‚10732.46044921875& disk.nvme0c0n1.out‚42286.404296875# proc.memory.rssMB‚ 3016.47265625 cpu‚25.680558791539752 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚54005.82958984375 disk.nvme0c0n1.in‚ 0.00390625% disk./.usageGB‚1166.9069633483887 disk.nvme1c1n1.in‚0% disk.nvme1c1n1.out‚22900.93359375# disk.nvme1n1.out‚22900.93359375p6µžã‚(:“ ¶ŠöÒ‹Ú²Ì gpu.0.gpu‚83.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚289.172' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.39066666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ÓÔä‚(:’ ÅŠöÒäijË gpu.0.gpu‚18.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚288.958' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.31933333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½+èØZå‚(:Ï ÅŠöÒ• Ì network.recv‚ 1.88254e+06 disk.nvme0c0n1.in‚ 0.00390625# disk.nvme0n1.out‚42950.07421875' disk.nvme1c1n1.out‚23111.4541015625) proc.memory.percent‚18.41108798980713% disk./.usageGB‚1166.8607063293457 disk.dm-1.in‚0# disk.dm-1.out‚10732.54248046875 disk.rbd0.in‚0% disk.nvme0c0n1.out‚42950.07421875% disk.nvme1n1.out‚23111.4541015625$ memory_percent‚97.01614379882812 cpu‚25.680137808125746 network.sent‚ 1.2367662e+07 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚54878.67041015625 disk.nvme1n1.in‚0# proc.memory.availableMB‚488.875# proc.memory.rssMB‚ 3016.47265625 proc.cpu.threads‚17) disk./.usagePercent‚76.00460552544183 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 210.59375G¾Ø-]æ‚(:Ò ÔŠöÒªõ¹Ë) disk./.usagePercent‚76.00457670245646 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚54879.21337890625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚23112.33544921875# proc.memory.availableMB‚488.875 disk.nvme1n1.in‚0 disk.md0.out‚ 211.11328125 disk.dm-1.in‚0 disk.dm-1.out‚10732.74609375& disk.nvme0c0n1.out‚42951.052734375$ disk.nvme0n1.out‚42951.052734375$ memory_percent‚97.01614379882812) proc.memory.percent‚18.41108798980713 network.sent‚ 1.2372503e+07$ disk./.usageGB‚1166.860263824463& disk.nvme1n1.out‚23112.33544921875 disk.nvme1c1n1.in‚0 cpu‚25.67965425743891 proc.cpu.threads‚17 network.recv‚ 1.886049e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625# proc.memory.rssMB‚ 3016.47265625í»ˆžç‚(:“ ÔŠöÒÙ¥€Ì gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚255.058' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.01933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+’"è- BQ+_wandbruntime‚7054  _runtime‚7054}…ýÇéÁ % _timestamp‚1.7845138811527326e+09 _runtime‚7054.728338019 _step‚639 & lr-AdamW/pg1‚0.0004225891596013432 & lr-AdamW/pg2‚0.0004225891596013432 trainer/global_step‚7799ÄÌÌÌêÆ trainer/global_step‚7799 % _timestamp‚1.7845138811527326e+09 _runtime‚7054.728338019 _step‚639 & lr-AdamW/pg1‚0.0004225891596013432 & lr-AdamW/pg2‚0.0004225891596013432ÿ8&>2ë- _wandbruntime‚7054  _runtime‚7054Iݾ·ù'ìóO ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1193481225406976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9055558444544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9041893831156 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63122657 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18642707 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13609017 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32251437 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1202536783851520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13609020 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚268470082159616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1186498633572432 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7845138811539235e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32251727 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9106750977536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚268470206041088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1193481225406976 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1186498616533072 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18642420 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9042234392430 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9106856246784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7054.729538751 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1202536783851520 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1193481242446336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1202537142315008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1193481242446336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9055558444544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1195540867964862 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77151382 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77161525 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63122655 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9055899868672 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13609017 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32251437 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1202536783851520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13609020 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚268470082159616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1186498633572432 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7845138811539235e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32251727 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9106750977536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚268470206041088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1193481225406976 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1186498616533072 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18642420 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9042234392430 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9106856246784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7054.729538751 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1202536783851520 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1193481242446336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1202537142315008€Rªíó2î- _wandbruntime‚7055  _runtime‚7055$š°}'ïýM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚259396530064384 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140291486 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77161038 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77161038 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1193629499909120 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77171195 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13610754 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚259396511452160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9056668543488 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1193629499909120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140301645 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚7055.353540187 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1202686526923264 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1193629516948480 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1195689049419952 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1186646047200544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18644664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1195689407020698 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9043342780794 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18644954 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32255415 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1202686168452608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9043002219408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9057009974784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9056668543488 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 % _timestamp‚1.7845138817777734e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63130450 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63130450 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77171195 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63130448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _step‚641 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32255708 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140301645 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚7055.353540187 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1202686526923264 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1193629516948480 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1195689049419952 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1186646047200544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18644664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1195689407020698 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9043342780794 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18644954 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32255415 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1202686168452608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9043002219408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9057009974784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9056668543488 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 % _timestamp‚1.7845138817777734e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63130450 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63130450 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77171195 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63130448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _step‚641 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32255708 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1210006078060544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141158484 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77636034 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1200894950514176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32452211 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13695717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63512307 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚270185059067392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1202967884551870 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1193870149906672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77646177 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13695720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚7849 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚270184928894464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261022344686080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845139247846687e+09 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32452497 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18756777 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1200894967553536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1210006436524032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9111127546368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63512307 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9162584208384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9111468970496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9097717605838 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1202967526951236 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9111127546368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63512305 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9162695769088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9097377044564 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77636034 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1193870166946032 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 _step‚644 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63512305 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9111468970496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚261022363298304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18756494 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚7098.360163746 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141148339 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1200894950514176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1210006078060544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141158484 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77636034 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1200894950514176 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32452211 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13695717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63512307 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚270185059067392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1202967884551870 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1193870149906672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77646177 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13695720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚7849 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚270184928894464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261022344686080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845139247846687e+09 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32452497 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18756777 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1200894967553536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1210006436524032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9111127546368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63512307 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9162584208384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9111468970496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9097717605838 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1202967526951236 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9111127546368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63512305 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9162695769088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9097377044564 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77636034 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1193870166946032 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 _step‚644„%8µL2€- _wandbruntime‚7098  _runtime‚7098Õ+³'ýM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1194017597613504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18758970 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7845139253993993e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9112237645312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63520098 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77655847 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9163810996224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13697454 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1194017580574144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77645690 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141165788 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚7098.975270989 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚270219222301696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261055543568384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63520100 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9098825994202 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1210155462661632 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32456134 trainer/global_step‚7849 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141175947 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63520100 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1210155821132288 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1201043225016320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1201043225016320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13697451 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9112237645312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141175947 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚645 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9112579076608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360bÆX5'‚‚N N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13697454 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1194017580574144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77645690 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141165788 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚7098.975270989 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚270219222301696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261055543568384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63520100 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9098825994202 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1210155462661632 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32456134 trainer/global_step‚7849 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141175947 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63520100 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1210155821132288 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1201043225016320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1201043225016320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13697451 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9112237645312 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141175947 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚645 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9112579076608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1194017597613504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18758970 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7845139253993993e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9112237645312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63520098 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77655847 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9163810996224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184…QO,2ƒ- _wandbruntime‚7098  _runtime‚7098é[G› „… 2loss/SRqqt2/assignment_loss‚0.5540098807646835 2loss/FBt2/assignment_loss‚0.004489819841083088 /loss/FBt2/detection_loss‚0.7633281083197347 %loss/total_loss‚8.859589953303598 % _timestamp‚1.7845139254013944e+09 0loss/FRt2/assignment_loss‚0.9023426627600065 /loss/FRt2/detection_loss‚1.5701437492083798 1loss/SRqqt1/detection_loss‚0.7977697841543441 trainer/global_step‚7849 .loss/FBt1/detection_loss‚0.767456837289501 2loss/SRqqt1/assignment_loss‚0.6049189979270517 3loss/FBt1/assignment_loss‚0.0010715358388758195 0loss/FRt1/detection_loss‚0.49635418120929636 1loss/SRqqt2/detection_loss‚0.6740343944799716 epoch‚19 _runtime‚7098.976780189 _step‚646 0loss/FRt1/assignment_loss‚1.7236700015106703ö¼oª…Š 1loss/SRqqt1/detection_loss‚0.7977697841543441 trainer/global_step‚7849 .loss/FBt1/detection_loss‚0.767456837289501 2loss/SRqqt1/assignment_loss‚0.6049189979270517 3loss/FBt1/assignment_loss‚0.0010715358388758195 0loss/FRt1/detection_loss‚0.49635418120929636 1loss/SRqqt2/detection_loss‚0.6740343944799716 epoch‚19 _runtime‚7098.976780189 _step‚646 0loss/FRt1/assignment_loss‚1.7236700015106703 2loss/SRqqt2/assignment_loss‚0.5540098807646835 2loss/FBt2/assignment_loss‚0.004489819841083088 /loss/FBt2/detection_loss‚0.7633281083197347 %loss/total_loss‚8.859589953303598 % _timestamp‚1.7845139254013944e+09 0loss/FRt2/assignment_loss‚0.9023426627600065 /loss/FRt2/detection_loss‚1.5701437492083798†r F.f†‚(:Û ‹öÒÕæÓË disk.nvme1c1n1.in‚0 disk.md0.in‚0" disk.dm-0.out‚54895.3642578125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚42967.53369140625( disk.nvme1c1n1.out‚23117.29736328125" memory_percent‚97.019362449646( proc.memory.availableMB‚ 488.34765625 network.sent‚ 1.2461364e+07) disk./.usagePercent‚76.00452352901796 disk.nvme1n1.in‚0 disk.nvme0c0n1.in‚ 0.00390625 cpu‚25.680595809405588 network.recv‚ 1.900698e+06 disk.dm-0.in‚ 0.00390625# proc.memory.rssMB‚ 3016.47265625) proc.memory.percent‚18.41108798980713 proc.cpu.threads‚17$ disk./.usageGB‚1166.859447479248& disk.nvme1n1.out‚23117.29736328125 disk.md0.out‚ 213.45703125 disk.dm-1.in‚0" disk.dm-1.out‚10733.1279296875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚42967.53369140625tB쑇‚(:† ‹öÒåÄ“Ì gpu.0.gpu‚58.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚202.53' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.51 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09fýâiˆ‚(:Þ Ÿ‹öÒõ©ÅË disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625) disk./.usagePercent‚76.00456403028187" disk.dm-0.out‚54897.6396484375 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚42970.12255859375( disk.nvme1c1n1.out‚23118.09521484375) proc.memory.percent‚18.41108798980713 cpu‚25.680987493660307 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚0( disk.nvme0c0n1.out‚42970.12255859375 disk.nvme1c1n1.in‚0$ memory_percent‚97.01964855194092( proc.memory.availableMB‚ 488.30078125 disk.md0.out‚ 213.94921875" disk.dm-1.out‚10733.1748046875# proc.memory.rssMB‚ 3016.47265625 network.sent‚ 1.2510976e+07 network.recv‚ 1.905791e+06% disk./.usageGB‚1166.8600692749023& disk.nvme1n1.out‚23118.09521484375 disk.md0.in‚0 disk.dm-0.in‚ 0.003906258^1 ž‰‚(:“ Ÿ‹öÒ¡Ë‡Ì gpu.0.gpu‚72.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚122.887' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.96233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ã ËgŠ‚(:Ü ®‹öÒÐ’ŸË disk.rbd0.out‚0 network.sent‚ 1.2513898e+07& disk.nvme1n1.out‚23119.06494140625 disk.md0.out‚ 214.61328125% disk.nvme0n1.out‚42971.3154296875( disk.nvme1c1n1.out‚23119.06494140625) proc.memory.percent‚18.41108798980713" disk.dm-0.out‚54898.2763671875 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚42971.3154296875 disk.nvme0n1.in‚ 0.00390625 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0# disk.dm-1.out‚10733.29931640625 network.recv‚ 1.908957e+06) disk./.usagePercent‚76.00456974518414% disk./.usageGB‚1166.8601570129395 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0$ memory_percent‚97.02143669128418' proc.memory.availableMB‚ 488.0078125# proc.memory.rssMB‚ 3016.47265625 cpu‚25.681344011669868q!œž‹‚(:“ ®‹öÒƒ‰íË gpu.0.gpu‚72.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.331' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.11033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 HT72Œ- _wandbruntime‚7141  _runtime‚7141˜&¿SÉà ' lr-AdamW/pg1‚0.00041801006431411346 ' lr-AdamW/pg2‚0.00041801006431411346 trainer/global_step‚7899 % _timestamp‚1.7845139679501266e+09 _runtime‚7141.525803913 _step‚647uXÿÎŽÈ _runtime‚7141.525803913 _step‚647 ' lr-AdamW/pg1‚0.00041801006431411346 ' lr-AdamW/pg2‚0.00041801006431411346 trainer/global_step‚7899 % _timestamp‚1.7845139679501266e+09‡y?Í}2- _wandbruntime‚7141  _runtime‚7141[Tp!æóO ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1208308675621376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚271899910552576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78120686 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1217475730733056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9166696648192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63901955 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9153200819246 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9167038072320 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 _step‚648 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1208308692660736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9152860257972 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18871421 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚262681376802304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚262681358190080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63901957 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32654114 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚271899790865408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1201241683280272 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1217475372269568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9218533750272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1217475372269568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚7141.528105317 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78130829 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142022641 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13782417 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7845139679524126e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18871694 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63901957 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63901955 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1217475730733056 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9167038072320 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32653838 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚7899 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78120686 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9218432675328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1201241700319632 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1208308675621376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9166696648192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32653838 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚7899 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78120686 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9218432675328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1201241700319632 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1208308675621376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9166696648192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1217475372269568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9218533750272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1217475372269568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚7141.528105317 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78130829 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142022641 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13782417 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7845139679524126e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18871694 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232ˆö ž­2’- _wandbruntime‚7142  _runtime‚7142Òÿã~'“ýM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9167806747136 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1208456967162880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18873654 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1201389130987104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142040090 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1208456950123520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13784151 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1201389113947744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78140499 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142050249 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78140499 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18873946 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1210543440194714 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚649 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚271934084976128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9168148178432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9219546515968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78130342 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚262714538460160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1208456950123520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _runtime‚7142.141013748 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚262714557072384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142050249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63909750 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63909748 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63909748 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1217625115341312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142040090 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9153968646224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9168148178432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9167806747136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63909750 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78130342 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271934204656128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚7899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1217625115341312 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1217624756870656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32657805 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13784154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32658100 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452ø36—”‚N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1201389113947744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78140499 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142050249 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78140499 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18873946 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1210543440194714 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚649 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚271934084976128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9168148178432 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9219546515968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78130342 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚262714538460160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1208456950123520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _runtime‚7142.141013748 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚262714557072384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142050249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63909750 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63909748 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63909748 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1217625115341312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142040090 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.curre´àcq nt‚295 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9153968646224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9168148178432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9167806747136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63909750 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78130342 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271934204656128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚7899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1217625115341312 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1217624756870656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32657805 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13784154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32658100 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9167806747136 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1208456967162880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18873654 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1201389130987104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142040090 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1208456950123520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13784151‰ Êô·2•- _wandbruntime‚7142  _runtime‚7142ç2¦ã –ƒ _step‚650 2loss/FBt1/assignment_loss‚0.017716583305695366 trainer/global_step‚7899 /loss/FRt1/assignment_loss‚1.874140354859253 3loss/FBt2/assignment_loss‚0.0024730096724290716 1loss/SRqqt1/detection_loss‚0.8616966012756165 /loss/FBt2/detection_loss‚0.2995777118811384 0loss/FRt2/assignment_loss‚0.7063465784435813 2loss/SRqqt1/assignment_loss‚0.5408853330861347 2loss/SRqqt2/assignment_loss‚0.5226937804919487 /loss/FRt1/detection_loss‚0.4698650671326829 .loss/FRt2/detection_loss‚1.412272078423904 /loss/FBt1/detection_loss‚0.7080972232634667 epoch‚19 % _timestamp‚1.7845139685712233e+09 1loss/SRqqt2/detection_loss‚0.6089769645027452 %loss/total_loss‚8.024741286338596 _runtime‚7142.146660133Qmåâ—ˆ epoch‚19 % _timestamp‚1.7845139685712233e+09 1loss/SRqqt2/detection_loss‚0.6089769645027452 %loss/total_loss‚8.024741286338596 _runtime‚7142.146660133 _step‚650 2loss/FBt1/assignment_loss‚0.017716583305695366 trainer/global_step‚7899 /loss/FRt1/assignment_loss‚1.874140354859253 3loss/FBt2/assignment_loss‚0.0024730096724290716 1loss/SRqqt1/detection_loss‚0.8616966012756165 /loss/FBt2/detection_loss‚0.2995777118811384 0loss/FRt2/assignment_loss‚0.7063465784435813 2loss/SRqqt1/assignment_loss‚0.5408853330861347 2loss/SRqqt2/assignment_loss‚0.5226937804919487 /loss/FRt1/detection_loss‚0.4698650671326829 .loss/FRt2/detection_loss‚1.412272078423904 /loss/FBt1/detection_loss‚0.7080972232634667ŠÝi1eb˜‚(:× ½‹öÒŽïÎË% disk.nvme0c0n1.out‚42973.18359375$ memory_percent‚96.55802249908447 network.recv‚ 1.912668e+06) disk./.usagePercent‚76.00457123602821 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 215.0078125 disk.rbd0.out‚0( proc.memory.availableMB‚ 563.93359375 network.sent‚ 1.2532743e+07 disk.nvme1n1.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.275166511535645 cpu‚25.68162426935984# disk.nvme0n1.out‚42973.18359375# disk.dm-1.out‚10733.40869140625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625# proc.memory.rssMB‚ 2830.36328125 proc.cpu.threads‚17$ disk./.usageGB‚1166.860179901123 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚23119.76513671875& disk.nvme1n1.out‚23119.76513671875 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚54899.8818359375”š€ž™‚(:“ ½‹öÒåÆ’Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚290.652' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.88399999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ëÍ^š‚(:Ó Ì‹öÒÄä«Ë disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚42974.06787109375& disk.nvme0n1.out‚42974.06787109375 disk.nvme1n1.in‚0% disk./.usageGB‚1166.6242942810059 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚54900.4912109375 disk.nvme1c1n1.in‚0" disk.nvme1n1.out‚ 23120.4296875# proc.memory.rssMB‚ 2830.36328125* proc.memory.percent‚17.275166511535645 network.recv‚ 1.917765e+06 disk.nvme0n1.in‚ 0.00390625$ disk.nvme1c1n1.out‚ 23120.4296875$ memory_percent‚96.55799865722656$ proc.memory.availableMB‚563.9375 cpu‚25.681908202294128 proc.cpu.threads‚17( disk./.usagePercent‚75.9892065970367 disk.md0.out‚ 215.37890625 disk.rbd0.out‚0 network.sent‚ 1.2581485e+07 disk.dm-1.in‚0# disk.dm-1.out‚10733.53271484375A ד›‚(:ˆ Ì‹öÒƒ€Ì gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚273.114' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.038 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÛmm^œ‚(:Ó Û‹öÒ«Î®Ë disk.md0.out‚ 215.578125 disk.rbd0.in‚0& disk.nvme0n1.out‚42979.19580078125$ disk.nvme1n1.out‚23120.916015625 disk.dm-0.out‚ 54905.6171875# disk.dm-1.out‚10733.57177734375$ proc.memory.availableMB‚563.9375* proc.memory.percent‚17.275166511535645% disk./.usageGB‚1166.6239433288574 disk.nvme1c1n1.in‚0$ memory_percent‚96.55799865722656 proc.cpu.threads‚17) disk./.usagePercent‚75.98918373742761 disk.nvme0n1.in‚ 0.00390625& disk.nvme1c1n1.out‚23120.916015625 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0# proc.memory.rssMB‚ 2830.36328125 cpu‚25.681806842176872 network.sent‚ 1.2588166e+07 network.recv‚ 1.921295e+06 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚42979.19580078125 disk.nvme1n1.in‚0 disk.md0.in‚0Ú²D‘ž‚(:“ Û‹öÒâöµÍ gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚261.734' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.24466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–¡¸2ž- _wandbruntime‚7185  _runtime‚7185'9gÇŸÁ trainer/global_step‚7949 % _timestamp‚1.7845140120376265e+09 _runtime‚7185.613272561 _step‚651 & lr-AdamW/pg1‚0.0004157097716497184 & lr-AdamW/pg2‚0.0004157097716497184€¶Þ„Ì Æ & lr-AdamW/pg2‚0.0004157097716497184 trainer/global_step‚7949 % _timestamp‚1.7845140120376265e+09 _runtime‚7185.613272561 _step‚651 & lr-AdamW/pg1‚0.0004157097716497184‹µo(q2¡- _wandbruntime‚7185  _runtime‚7185Î^˜Dø'¢òO aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1208613233693232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1217821917725886 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78605338 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264340390306304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 $ _timestamp‚1.784514012038756e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1224944666478592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9222607174144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9222607174144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9222265750016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1215722400728576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1217821560125252 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32855027 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264340371694080 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1208613216653872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64291605 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142907088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9208343471380 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9222265750016 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0"±cý'£÷O eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264340390306304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 $ _timestamp‚1.784514012038756e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1224944666478592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9222607174144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9222607174144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9222265750016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1215722400728576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1217821560125252 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32855027 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264340371694080 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1208613216653872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64291605 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142907088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9208343471380 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9222265750016 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1208613233693232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1217821917725886 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78605338 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452ŒYê#˜2¤- _wandbruntime‚7186  _runtime‚7186Tÿ)½ ¥üM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264373551964160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1217970456781722 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142914392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9209451859632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7845140126588037e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1215870692270080 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64299398 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 1ÌSa²E15485696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64299400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1215870675230720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9223375848960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13870851 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚273648958563328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1208760664360704 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64299400 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78614994 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1225094409550336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚273649074049024 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78614994 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1225094051079680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32859041 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9223375848960 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142924551 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1208760647321344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264373570576384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚7186.234605506 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64299398 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64299400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1215870675230720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9223375848960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13870851 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚273648958563328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1208760664360704 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64299400 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78614994 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1225094409550336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚273649074049024 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78614994 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1225094051079680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32859041 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9223375848960 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142924551 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1208760647321344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264373570576384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚7186.234605506 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9209451859632 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7845140126588037e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511-ºÒ2§- _wandbruntime‚7186  _runtime‚7186LïÀݨ %loss/total_loss‚8.698800638503954 0loss/FRt1/assignment_loss‚1.8413077983860822 1loss/SRqqt2/assignment_loss‚0.606163193359173 /loss/FRt2/detection_loss‚1.5743897629998822 0loss/SRqqt1/detection_loss‚0.762534010966479 /loss/FBt2/detection_loss‚0.6195817093742362 trainer/global_step‚7949 0loss/FRt2/assignment_loss‚0.9057949614894892 1loss/SRqqt1/assignment_loss‚0.615170516876124 1loss/FBt2/assignment_loss‚0.01299420669024378 % _timestamp‚1.7845140126598377e+09 _runtime‚7186.235363937 1loss/SRqqt2/detection_loss‚0.6834440657234553 epoch‚19 _step‚654 2loss/FBt1/assignment_loss‚0.012541071570101096 0loss/FRt1/detection_loss‚0.48022404004084523 /loss/FBt1/detection_loss‚0.5846553010278426o³f' ©† 1loss/SRqqt2/detection_loss‚0.6834440657234553 epoch‚19 _step‚654 2loss/FBt1/assignment_loss‚0.012541071570101096 0loss/FRt1/detection_loss‚0.48022404004084523 /loss/FBt1/detection_loss‚0.5846553010278426 %loss/total_loss‚8.698800638503954 0loss/FRt1/assignment_loss‚1.8413077983860822 1loss/SRqqt2/assignment_loss‚0.606163193359173 /loss/FRt2/detection_loss‚1.5743897629998822 0loss/SRqqt1/detection_loss‚0.762534010966479 /loss/FBt2/detection_loss‚0.6195817093742362 trainer/global_step‚7949 0loss/FRt2/assignment_loss‚0.9057949614894892 1loss/SRqqt1/assignment_loss‚0.615170516876124 1loss/FBt2/assignment_loss‚0.01299420669024378 % _timestamp‚1.7845140126598377e+09 _runtime‚7186.235363937Žwýt©aª‚(:Ö ê‹öÒߪñË disk.nvme1c1n1.in‚0 disk.md0.out‚ 216.234375 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚ 54906.2890625 network.sent‚ 1.260748e+07 network.recv‚ 1.93159e+06 disk.dm-1.in‚0" disk.dm-1.out‚10733.6806640625 disk.rbd0.in‚0& disk.nvme1n1.out‚23121.83837890625# proc.memory.rssMB‚ 2830.36328125* proc.memory.percent‚17.275166511535645 disk.rbd0.out‚0& disk.nvme0n1.out‚42980.45458984375 disk.nvme1n1.in‚0 disk.md0.in‚0$ memory_percent‚96.55561447143555& proc.memory.availableMB‚ 564.328125 cpu‚25.681148046363454 proc.cpu.threads‚17( disk.nvme0c0n1.out‚42980.45458984375 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚23121.83837890625) disk./.usagePercent‚75.98918398590162% disk./.usageGB‚1166.6239471435547 disk.nvme0c0n1.in‚ 0.00390625k뀓«‚(:ˆ ê‹öÒöçºÌ gpu.0.gpu‚50.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.311' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.437 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ']éc¬‚(:Ø ù‹öÒ»ØÍË network.sent‚ 1.265897e+07 disk.md0.out‚ 216.76171875 disk.dm-1.in‚0 cpu‚25.68067049566643 proc.cpu.threads‚17 network.recv‚ 1.936708e+06% disk./.usageGB‚1166.9032707214355 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚43034.03173828125& disk.nvme1c1n1.out‚23139.181640625 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.rbd0.out‚0( disk./.usagePercent‚76.0073779984651 disk.nvme1c1n1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚43034.03173828125$ memory_percent‚96.55568599700928( proc.memory.availableMB‚ 564.31640625* proc.memory.percent‚17.275190353393555$ disk.nvme1n1.out‚23139.181640625 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚54975.96826171875" disk.dm-1.out‚10733.8056640625 disk.nvme0c0n1.in‚ 0.00390625" proc.memory.rssMB‚ 2830.3671875\Cž­‚(:“ ù‹öÒšÇ–Ì gpu.0.gpu‚98.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.444' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.14800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØJ¢2®- _wandbruntime‚7228  _runtime‚7228¥‚?ÛǯÁ & lr-AdamW/pg2‚0.0004134025124771574 trainer/global_step‚7999 % _timestamp‚1.7845140550742774e+09 _runtime‚7228.649803247 _step‚655 & lr-AdamW/pg1‚0.0004134025124771574yGñÚÌ°Æ & lr-AdamW/pg1‚0.0004134025124771574 & lr-AdamW/pg2‚0.0004134025124771574 trainer/global_step‚7999 % _timestamp‚1.7845140550742774e+09 _runtime‚7228.649803247 _step‚655@:!2±- _wandbruntime‚7228  _runtime‚7228K…ø'²òO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64681255 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚265999403810304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9330228126720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64681255 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143781390 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1225248576712260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1232414319151104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33056452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚275329497569792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143771245 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79089990 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9264167246062 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚7999 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33056156 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1223136142875136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 _step‚656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19100632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1232413960687616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344HÓö"³÷O XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current¤<¬Û$‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9264167246062 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚7999 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33056156 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1223136142875136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 _step‚656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19100632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1232413960687616 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64681255 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚265999403810304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9330228126720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64681255 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143781390 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1225248576712260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1232414319151104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33056452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚275329497569792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143771245 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79089990 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ¨[ì2´- _wandbruntime‚7229  _runtime‚7229è¼F†'µýM F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845140556997082e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1216132197734304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚275363925173248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1232563703759360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9279286382080 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1232563703759360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266032584080384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143798853 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9278944950784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64689048 trainer/global_step‚7999 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9331229539328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1225397473368730 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1223284417377280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266032565468160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79099646 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13957554 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64689050 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19102903 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1216132180694944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33060146 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9331341092864 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79109803 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143798853 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79099646 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79109803 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1223284400337920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19102595 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚275363795007488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1232563345288704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64689050 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚311 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚657 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚308 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9265275634426 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9264935073040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528Ú)Š'¶‚N T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1232563345288704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64689050 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚311 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚657 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚308 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9265275634426 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9264935073040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845140556997082e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1216132197734304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚275363925173248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1232563703759360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9279286382080 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1232563703759360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266032584080384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143798853 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9278944950784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64689048 trainer/global_step‚7999 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9331229539328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1225397473368730 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1223284417377280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266032565468160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79099646 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13957554 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64689050 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19102903 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1216132180694944 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33060146 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9331341092864 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79109803 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143798853 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79099646 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79109803 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1223284400337920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19102595 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚275363795007488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0‘xߦ2·- _wandbruntime‚7229  _runtime‚7229¿(€l ¸ƒ 1loss/SRqqt1/detection_loss‚0.6933196988347968 /loss/FBt1/detection_loss‚0.4535165130327553 1loss/SRqqt2/detection_loss‚0.6852079630588888 trainer/global_step‚7999 % _timestamp‚1.7845140557007816e+09 %loss/total_loss‚8.400420930424335 0loss/FRt2/assignment_loss‚0.8329507186454224 2loss/FBt2/assignment_loss‚0.009029170427255942 .loss/FBt2/detection_loss‚0.718376456031528 _runtime‚7229.276084483 2loss/SRqqt1/assignment_loss‚0.5457868591874717 2loss/FBt1/assignment_loss‚0.003355937993353394 /loss/FRt2/detection_loss‚1.6670145893182893 _step‚658 epoch‚19 0loss/FRt1/assignment_loss‚1.7573423435950415 2loss/SRqqt2/assignment_loss‚0.5491022946253411 /loss/FRt1/detection_loss‚0.4854183856741915ÅñWº¹ˆ epoch‚19 0loss/FRt1/assignment_loss‚1.7573423435950415 2loss/SRqqt2/assignment_loss‚0.5491022946253411 /loss/FRt1/detection_loss‚0.4854183856741915 1loss/SRqqt1/detection_loss‚0.6933196988347968 /loss/FBt1/detection_loss‚0.4535165130327553 1loss/SRqqt2/detection_loss‚0.6852079630588888 trainer/global_step‚7999 % _timestamp‚1.7845140557007816e+09 %loss/total_loss‚8.400420930424335 0loss/FRt2/assignment_loss‚0.8329507186454224 2loss/FBt2/assignment_loss‚0.009029170427255942 .loss/FBt2/detection_loss‚0.718376456031528 _runtime‚7229.276084483 2loss/SRqqt1/assignment_loss‚0.5457868591874717 2loss/FBt1/assignment_loss‚0.003355937993353394 /loss/FRt2/detection_loss‚1.6670145893182893 _step‚658’;T× lº‚(:á ˆŒöÒ’ëüË disk.md0.out‚ 216.91796875 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2830.37890625 cpu‚25.680328260168533 network.sent‚ 1.2674836e+07% disk./.usageGB‚1166.9036102294922 disk.nvme1n1.in‚0& disk.nvme1n1.out‚23139.55615234375 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0( proc.memory.availableMB‚ 564.05078125* proc.memory.percent‚17.275261878967285 network.recv‚ 1.939923e+06) disk./.usagePercent‚76.00740011265216( disk.nvme0c0n1.out‚43086.04736328125& disk.nvme0n1.out‚43086.04736328125( disk.nvme1c1n1.out‚23139.55615234375 disk.md0.in‚0# disk.dm-0.out‚55027.93701171875# disk.dm-1.out‚10733.87548828125 disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚96.55730724334717 proc.cpu.threads‚17‘¶&»‚(:’ ˆŒöÒ½¸¿Ì gpu.0.gpu‚21.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.965' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.98833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àpŠ¥P¼‚(:Ô —ŒöÒ³˜¯Ë disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625( disk./.usagePercent‚76.0047548583229 disk.md0.in‚0# disk.dm-0.out‚55163.59912109375' disk.nvme0c0n1.out‚43206.0595703125 disk.nvme1c1n1.in‚0! disk.nvme1n1.out‚ 23156.390625$ m‚þ«¯emory_percent‚97.24442958831787 cpu‚25.685675550862385 disk.md0.out‚ 217.41015625% disk.nvme0n1.out‚43206.0595703125# disk.nvme1c1n1.out‚ 23156.390625# proc.memory.rssMB‚ 2942.84765625 proc.cpu.threads‚17 network.recv‚ 1.944937e+06 disk.dm-1.in‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 451.47265625* proc.memory.percent‚17.961716651916504 network.sent‚ 1.2724458e+07% disk./.usageGB‚1166.8629989624023 disk.dm-0.in‚ 0.00390625# disk.dm-1.out‚10734.00830078125²tF.ž½‚(:“ —ŒöÒ¯ûœÌ gpu.0.gpu‚75.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.108' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.36933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õNšò2¾- _wandbruntime‚7245  _runtime‚7245ó,%Äa=¿Ûz 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2721916139125824 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988068342208862 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6300357580184937 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24361398816108704 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1675117164850235 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.8160386681556702 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949959516525269 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7689179182052612 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2722999155521393 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 )jet/accuracy_1_of_4‚0.981353759765625 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.31956425309181213 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.781863808631897 particle/accuracy_2_of_2‚1 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.336927205324173 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7643883228302002 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.703111469745636 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/sensitivity‚0.6489822268486023 )loss/val_total_loss‚8.007595637061488 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956770539283752 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6687224507331848 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6576963663101196 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.746705949306488 'particle/f_score‚0.6636024117469788 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 epoch‚19 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.490778386592865 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +jet/accuracy_3_of_3‚0.07672887295484543 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.054859962314367294 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9470755457878113 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7516017556190491 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.702079176902771 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.776611864566803 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6681392192840576 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.812842607498169 +jet/accuracy_1_of_1‚0.36763256788253784 6"Purity/*FRt0SRqqt1FBt/event_purity‚0.9081991314888 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6751973032951355 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983391761779785 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.33320313692092896 (particle/accuracy‚0.8587162494659424 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8868104219436646 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16840746998786926 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04481525719165802 % _timestamp‚1.7845140719878561e+09 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26198065280914307 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.86360102891922 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6774337291717529 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9468741416931152 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2496969848871231 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5824358463287354 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0566520020365715 7"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.86360102891922 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46824344992637634 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31922727823257446 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5674999952316284 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.712230384349823 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30954501032829285 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6719855070114136 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8878496289253235 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963415265083313 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7147404551506042 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04578039050102234 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8846303224563599 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.061292972415685654 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2577570378780365 particle/accuracy_1_of_3‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6159359216690063 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23383283615112305 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22507883608341217 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974848628044128 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7470877766609192 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6730031967163086 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.840684175491333 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7700383067131042 +jet/accuracy_2_of_2‚0.16753381490707397 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 trainer/global_step‚7999 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3709411323070526 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6959941983222961 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7042067050933838 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8782586455345154 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.029763588681817055 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5789318084716797 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8755858540534973 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5053073167800903 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 +particle/specificity‚0.9160786867141724 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3632301390171051 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953127503395081 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45659270882606506 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.260628342628479 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6707785725593567 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06911180913448334 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 6validation_average_jet_accuracy‚0.4053523540496826 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965001940727234 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3201276361942291 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15338903665542603 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.12014035135507584 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9886590242385864 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7518632411956787 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2730841636657715 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3090569078922272 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.779808521270752 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18286411464214325 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 _runtime‚7245.564118253 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4025272727012634 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2651900351047516 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_3_of_3‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8135393857955933 jet/accuracy_1_of_5‚1 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.666289210319519 *jet/accuracy_1_of_3‚0.9367994666099548 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04611048847436905 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6875888109207153 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.32115596532821655 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981421232223511 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.755006730556488 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7663643956184387 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.13530339300632477 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6707785725593567 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 )jet/accuracy_2_of_3‚0.610088586807251 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3906250298023224 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4855436086654663 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05416092276573181 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6796875 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30081096291542053 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7454329133033752 *jet/accuracy_2_of_4‚0.8260692358016968 particle/accuracy_5_of_5‚1 _step‚659 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9886862635612488 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3989511728286743 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7539307475090027 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 +jet/accuracy_3_of_4‚0.09514929354190826 ,jet/accuracy_4_of_4‚0.013899499550461769 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958755970001221 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974848628044128 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6955382823944092 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 jet/accuracy_4_of_5‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3159260153770447 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5201992988586426 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6823117733001709 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.029763588681817055 particle/accuracy_1_of_1‚1 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.220860555768013 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6535446047782898 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8922713994979858 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3090569078922272 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094§ñwßf=Ààz trainer/global_step‚7999 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3709411323070526 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6959941983222961 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7042067050933838 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8782586455345154 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.029763588681817055 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5789318084716797 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8755858540534973 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5053073167800903 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 +particle/specificity‚0.9160786867141724 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3632301390171051 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953127503395081 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45659270882606506 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.260628342628479 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6707785725593567 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06911180913448334 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 6validation_average_jet_accuracy‚0.4053523540496826 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965001940727234 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3201276361942291 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15338903665542603 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.12014035135507584 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9886590242385864 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7518632411956787 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2730841636657715 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3090569078922272 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.779808521270752 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18286411464214325 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 _runtime‚7245.564118253 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4025272727012634 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2651900351047516 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_3_of_3‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8135393857955933 jet/accuracy_1_of_5‚1 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.666289210319519 *jet/accuracy_1_of_3‚0.9367994666099548 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04611048847436905 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6875888109207153 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.32115596532821655 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981421232223511 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.755006730556488 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7663643956184387 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.13530339300632477 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6707785725593567 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 )jet/accuracy_2_of_3‚0.610088586807251 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3906250298023224 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4855436086654663 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05416092276573181 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6796875 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30081096291542053 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7454329133033752 *jet/accuracy_2_of_4‚0.8260692358016968 particle/accuracy_5_of_5‚1 _step‚659 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9886862635612488 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3989511728286743 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7539307475090027 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 +jet/accuracy_3_of_4‚0.09514929354190826 ,jet/accuracy_4_of_4‚0.013899499550461769 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9958755970001221 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974848628044128 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6955382823944092 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 jet/accuracy_4_of_5‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3159260153770447 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5201992988586426 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6823117733001709 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.029763588681817055 particle/accuracy_1_of_1‚1 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.220860555768013 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6535446047782898 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8922713994979858 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3090569078922272 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2721916139125824 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988068342208862 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6300357580184937 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24361398816108704 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1675117164850235 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.8160386681556702 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949959516525269 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7689179182052612 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2722999155521393 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 )jet/accuracy_1_of_4‚0.981353759765625 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.31956425309181213 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.781863808631897 particle/accuracy_2_of_2‚1 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.336927205324173 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7643883228302002 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.703111469745636 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/sensitivity‚0.6489822268486023 )loss/val_total_loss‚8.007595637061488 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956770539283752 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6687224507331848 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6576963663101196 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.746705949306488 'particle/f_score‚0.6636024117469788 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 epoch‚19 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.490778386592865 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +jet/accuracy_3_of_3‚0.07672887295484543 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.054859962314367294 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9470755457878113 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7516017556190491 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.702079176902771 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.776611864566803 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6681392192840576 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.812842607498169 +jet/accuracy_1_of_1‚0.36763256788253784 6"Purity/*FRt0SRqqt1FBt/event_purity‚0.9081991314888 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6751973032951355 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983391761779785 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.33320313692092896 (particle/accuracy‚0.8587162494659424 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8868104219436646 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.16840746998786926 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04481525719165802 % _timestamp‚1.7845140719878561e+09 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26198065280914307 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.86360102891922 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6774337291717529 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9468741416931152 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2496969848871231 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5824358463287354 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0566520020365715 7"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.86360102891922 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46824344992637634 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31922727823257446 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5674999952316284 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.712230384349823 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30954501032829285 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6719855070114136 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8878496289253235 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963415265083313 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7147404551506042 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04578039050102234 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8846303224563599 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.061292972415685654 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2577570378780365 particle/accuracy_1_of_3‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6159359216690063 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23383283615112305 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22507883608341217 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974848628044128 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7470877766609192 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6730031967163086 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.840684175491333 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7700383067131042 +jet/accuracy_2_of_2‚0.16753381490707397 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5“k {î0Á‚(:Ù ¦ŒöÒŠ‡üÌ disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625" proc.memory.rssMB‚ 2947.7734375) proc.memory.percent‚17.99178123474121 cpu‚25.68500620370002% disk./.usageGB‚1166.8630065917969 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚23GÑÂ4402.6904296875 disk.nvme1n1.in‚0% disk.nvme1n1.out‚23402.6904296875# disk.dm-0.out‚55164.24755859375 disk.dm-1.in‚0 network.recv‚ 1.948815e+06% disk.nvme0n1.out‚43207.1142578125 disk.nvme1c1n1.in‚0 disk.dm-0.in‚ 0.00390625# disk.dm-1.out‚10979.60498046875$ memory_percent‚95.48830986022949' proc.memory.availableMB‚ 739.1953125 proc.cpu.threads‚17 network.sent‚ 1.2730999e+07' disk.nvme0c0n1.out‚43207.1142578125 disk.md0.in‚0 disk.md0.out‚ 217.93359375) disk./.usagePercent‚76.00475535527093HÉ`‘‚(:† ¦ŒöÒæÞÊÍ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚273.21' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.07 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó0ÜQeÂ(:Ú µŒöÒª¿ÅË cpu‚25.68491027185161 network.recv‚ 1.959583e+06% disk./.usageGB‚1166.8630065917969# disk.dm-1.out‚10979.69482421875 disk.nvme1c1n1.in‚0' disk.nvme1c1n1.out‚23404.8623046875 disk.nvme1n1.in‚0) disk./.usagePercent‚76.00475535527093 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚55174.39013671875' disk.nvme0c0n1.out‚43216.5654296875 disk.md0.in‚0 disk.md0.out‚ 218.60546875( proc.memory.availableMB‚ 739.10546875 network.sent‚ 1.2791603e+07 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚95.48885822296143) proc.memory.percent‚17.99178123474121 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚43216.5654296875% disk.nvme1n1.out‚23404.8623046875" proc.memory.rssMB‚ 2947.7734375#I‰AžÄ‚(:“ µŒöÒ“ßÌ gpu.0.gpu‚91.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚260.838' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.94600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ͧ›;žÅ‚(:“ ÄŒöÒñàÙË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.955' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.65166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÑ&aÆ‚(:Ö ÄŒöÒ±¯ÎË' disk.nvme0c0n1.out‚43217.2451171875% disk./.usageGB‚1166.8630065917969 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0 disk.md0.out‚ 218.8203125 cpu‚25.685077610115883 network.sent‚ 1.2794525e+07% disk.nvme0n1.out‚43217.2451171875( disk.nvme1c1n1.out‚23650.91259765625# memory_percent‚95.4904556274414% proc.memory.availableMB‚ 738.84375" proc.memory.rssMB‚ 2947.7734375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 network.recv‚ 1.96276e+06) disk./.usagePercent‚76.00475535527093 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚55174.90966796875" disk.dm-1.out‚11225.4208984375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0& disk.nvme1n1.out‚23650.91259765625 disk.md0.in‚0) proc.memory.percent‚17.99178123474121‡Ágç2Ç- _wandbruntime‚7290  _runtime‚7290º  ÇÈÁ & lr-AdamW/pg1‚0.0004110884349873755 & lr-AdamW/pg2‚0.0004110884349873755 trainer/global_step‚8049 % _timestamp‚1.7845141164911716e+09 _runtime‚7290.066870101 _step‚660ÙâøµÌÉÆ _step‚660 & lr-AdamW/pg1‚0.0004110884349873755 & lr-AdamW/pg2‚0.0004110884349873755 trainer/global_step‚8049 % _timestamp‚1.7845141164911716e+09 _runtime‚7290.066870101”5+™2Ê- _wandbruntime‚7290  _runtime‚7290)éø'ËòO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65193713 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1232492211857920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14046633 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79656141 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33299850 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79656141 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 % _timestamp‚1.7845141164941874e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9354063216128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79666284 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65193715 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1241845916610560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1225277539409296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1241845916610560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1232492194818560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267680977364480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1232492211857920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1234617510757028 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33299562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19253214 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 _step‚661 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65193715 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚277087707404288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚8049 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚277087824994304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1241846275074048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267680995976704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9339613747098 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85".µ•ý'Ì÷O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14046633 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79656141 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33299850 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79656141 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 % _timestamp‚1.7845141164941874e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9354063216128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79666284 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65193715 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1241845916610560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1225277539409296 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1241845916610560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1232492194818560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267680977364480 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1232492211857920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1234617510757028 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33299562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19253214 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 _step‚661 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65193715 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚277087707404288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚8049 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚277087824994304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1241846275074048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267680995976704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9339613747098 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65193713 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1232492211857920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344•þôAƒ2Í- _wandbruntime‚7290  _runtime‚7290é^5¹ÎüM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65201508 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79665797 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144877462 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1225424987116128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚7290.698142569 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚277122120385024 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9407944138240 _step‚662 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚267714157634560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1241995659682304 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1232640486360064 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1234765692212118 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9355173322240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1232640486360064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144867303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79665797 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79675954 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33303604 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19255509 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19255237 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 % _timestamp‚1.7845141171223304e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9341062696736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14048370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚277122011190784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33303879 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.sm @|ÍIall_pool.freed‚ 9354831890944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9355173322240 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9354831890944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1232640469320704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109194240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9407853556224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚8049 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33303879 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9354831890944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9355173322240 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9354831890944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1232640469320704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109194240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9407853556224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚8049 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144877462 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1225424987116128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚7290.698142569 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚277122120385024 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9407944138240 _step‚662 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚267714157634560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1241995659682304 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1232640486360064 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1234765692212118 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9355173322240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1232640486360064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144867303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79665797 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79675954 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33303604 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19255509 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19255237 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 % _timestamp‚1.7845141171223304e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9341062696736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272–ܲe2Ð- _wandbruntime‚7290  _runtime‚7290ÅÌ Ñ„ /loss/FRt2/assignment_loss‚0.816129313616813 3loss/SRqqt1/assignment_loss‚0.49736479039105835 2loss/FBt1/assignment_loss‚0.004183544049367411 2loss/FBt2/assignment_loss‚0.002143922793831104 /loss/FBt2/detection_loss‚0.5920299131084572 .loss/FRt1/assignment_loss‚1.73709758057082 /loss/FRt2/detection_loss‚1.3862146058538338 1loss/SRqqt2/detection_loss‚0.6305811494373407 epoch‚20 _step‚663 0loss/FRt1/detection_loss‚0.49926669141572805 1loss/SRqqt1/detection_loss‚0.7422325638388949 0loss/FBt1/detection_loss‚0.40816371493459935 trainer/global_step‚8049 _runtime‚7290.699790591 2loss/SRqqt2/assignment_loss‚0.5660690553924356 %loss/total_loss‚7.881476845403179 % _timestamp‚1.7845141171243687e+09µ (Ò‰ /loss/FRt2/assignment_loss‚0.816129313616813 3loss/SRqqt1/assignment_loss‚0.49736479039105835 2loss/FBt1/assignment_loss‚0.004183544049367411 2loss/FBt2/assignment_loss‚0.002143922793831104 /loss/FBt2/detection_loss‚0.5920299131084572 .loss/FRt1/assignment_loss‚1.73709758057082 /loss/FRt2/detection_loss‚1.3862146058538338 1loss/SRqqt2/detection_loss‚0.6305811494373407 epoch‚20 _step‚663 0loss/FRt1/detection_loss‚0.49926669141572805 1loss/SRqqt1/detection_loss‚0.7422325638388949 0loss/FBt1/detection_loss‚0.40816371493459935 trainer/global_step‚8049 _runtime‚7290.699790591 2loss/SRqqt2/assignment_loss‚0.5660690553924356 %loss/total_loss‚7.881476845403179 % _timestamp‚1.7845141171243687e+09—(^P°eÓ‚(:Ú ÓŒöÒž©±Ë network.sent‚ 1.2799482e+07 network.recv‚ 1.966538e+06& disk.nvme1n1.out‚23652.00439453125 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚23652.00439453125$ memory_percent‚95.48919200897217 disk.md0.out‚ 219.48828125 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚43219.80224609375" proc.memory.rssMB‚ 2947.7734375 cpu‚25.685383055049794( disk./.usagePercent‚76.0047590823811 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.out‚55177.09765625" disk.dm-1.out‚11225.4716796875( disk.nvme0c0n1.out‚43219.80224609375 disk.nvme0n1.in‚ 0.00390625( proc.memory.availableMB‚ 739.05078125% disk./.usageGB‚1166.8630638122559 disk.nvme0c0n1.in‚ 0.00390625) proc.memory.percent‚17.99178123474121 proc.cpu.threads‚17Ihó“Ô‚(:ˆ ÓŒöÒÖɨÌ gpu.0.gpu‚64.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚270.315' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚90.105 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß/¤aÕ‚(:Ö âŒöÒºèÈË disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚ 55177.71875" proc.memory.rssMB‚ 2947.7734375) proc.memory.percent‚17.99178123474121 proc.cpu.threads‚17$ disk./.usageGB‚1166.862979888916 disk.rbd0.out‚0 disk.nvme1n1.in‚0) disk./.usagePercent‚76.00475361595285 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚43220.57763671875 disk.dm-1.in‚0! disk.dm-1.out‚11228.814453125( proc.memory.availableMB‚ 738.79296875 cpu‚25.686093128268073 network.sent‚ 1.2863624e+07 network.recv‚ 1.970563e+06 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚23655.76708984375& disk.nvme1n1.out‚23655.76708984375 disk.md0.in‚0$ memory_percent‚95.49076557159424( disk.nvme0c0n1.out‚43220.57763671875 disk.md0.out‚ 219.74609375 ¹×r‘Ö‚(:† âŒöÒÐé„Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.652' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌÜk2×- _wandbruntime‚7334  _runtime‚7334ã(òÉØà ' lr-AdamW/pg1‚0.00040876768780924574 ' lr-AdamW/pg2‚0.00040876768780924574 trainer/global_step‚8099 % _timestamp‚1.7845141613399365e+09 _runtime‚7334.915621609 _step‚664'ŽX²ÎÙÈ ' lr-AdamW/pg1‚0.00040876768780924574 ' lr-AdamW/pg2‚0.00040876768780924574 trainer/global_step‚8099 % _timestamp‚1.7845141613399365e+09 _runtime‚7334.915621609 _step‚664˜N('e2Ú- _wandbruntime‚7334  _runtime‚7334|Xø'ÛòO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9409290893824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19367871 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9395096960506 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚8099 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1249315569283072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784514161347188e+09 _runtime‚7334.922934871 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65583365 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80140793 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9395437521780 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65583363 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14133336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9462716138496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80140793 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1249315569283072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1232649089822256 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80150936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33501499 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9409632317952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14133333 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1239905919925760 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1239905936965120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚269340009480704 _step‚665 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19368163 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1249315210819584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9409632317952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145724156 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65583363 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1249315210819584 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1239905919925760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1232649072782896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1242044527344036 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145734301 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33501204 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚278802605932032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80150936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269339990868480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9409290893824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9462615063552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚269340009480704 _step‚665 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19368163 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1249315210819584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9409632317952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145724156 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65583363 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1249315210819584 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1239905919925760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1232649072782896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1242044527344036 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145734301 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33501204 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚278802605932032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80150936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269339990868480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9409290893824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9462615063552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1249315569283072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784514161347188e+09 _runtime‚7334.922934871 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65583365 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80140793 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9395437521780 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65583363 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14133336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9462716138496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80140793 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1249315569283072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1232649089822256 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80150936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33501499 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9409632317952™äÒk¸cÝ‚(:Ø ñŒöÒ£“ÂË) disk./.usagePercent‚75.98928163618824 disk.md0.in‚0 disk.md0.out‚ 220.16796875 disk.rbd0.out‚0( disk.nvme1c1n1.out‚23656.59619140625 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 738.08203125 proc.cpu.threads‚17 network.recv‚ 1.974173e+06 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0) proc.memory.percent‚17.99178123474121# disk./.usageGB‚1166.62544631958 disk.dm-1.in‚0! disk.dm-1.out‚11228.912109375' disk.nvme0c0n1.out‚43223.1123046875% disk.nvme0n1.out‚43223.1123046875$ memory_percent‚95.49510478973389" proc.memory.rssMB‚ 2947.7734375 network.sent‚ 1.2866546e+07 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚55180.08447265625& disk.nvme1n1.out‚23656.59619140625 cpu‚25.68670001878887¼Â€j“Þ‚(:ˆ ñŒöÒ‚£­Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.861' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.287 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºì¨2ß- _wandbruntime‚7335  _runtime‚73353‡‰ˆ'àûM XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1240054194427904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65591156 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19370133 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80160606 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80150449 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8099 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145751764 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1249464953891328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33505495 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1242193066399872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1240054211467264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65591158 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19370425 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9396545910144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65591158 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80150449 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9463727578112 $ _timestamp‚1.784514161970898e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9410742424064 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1232796503450368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9396205348758 _step‚666 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33505200 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145741605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80160606 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚269373171138560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9463837034496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65591156 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1249464953891328 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1240054194427904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9410400992768 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1242192708799126 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1240054211467264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚269373189750784 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1232796520489728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145751764 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278837026785280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145741605 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9410742424064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚278836898716672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1249464595420672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14135067 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85˜!mY'á€N E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9396545910144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65591158 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80150449 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9463727578112 $ _timestamp‚1.784514161970898e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9410742424064 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1232796503450368 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9396205348758 _step‚666 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33505200 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145741605 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80160606 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚269373171138560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9463837034496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65591156 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1249464953891328 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1240054194427904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9410400992768 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1242192708799126 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1240054211467264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚269373189750784 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1232796520489728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145751764 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278837026785280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145741605 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9410742424064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚278836898716672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1249464595420672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14135067 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1240054194427904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65591156 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19370133 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80160606 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80150449 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8099 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145751764 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1249464953891328 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33505495 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1242193066399872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1240054211467264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65591158 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19370425 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384šF,¤2â- _wandbruntime‚7335  _runtime‚7335­9;~㈠% _timestamp‚1.7845141619720407e+09 2loss/SRqqt2/assignment_loss‚0.5418589033560847 1loss/SRqqt2/detection_loss‚0.6190190817285469 0loss/FBt2/detection_loss‚0.43013447263898946 trainer/global_step‚8099 _step‚667 2loss/FBt1/assignment_loss‚0.005213656588454532 1loss/SRqqt1/detection_loss‚0.7498701162407031 /loss/FBt1/detection_loss‚0.6802012694614201 epoch‚20 _runtime‚7335.547394921 0loss/FRt1/assignment_loss‚1.7419501888763547 4loss/FBt2/assignment_loss‚0.00017634064550809588 0loss/FRt1/detection_loss‚0.46077456468057143 %loss/total_loss‚8.141083043431124 0loss/FRt2/assignment_loss‚0.7863800663424302 2loss/SRqqt1/assignment_loss‚0.5735467893179381 /loss/FRt2/detection_loss‚1.5519575935541237L}|ìä 0loss/FRt1/assignment_loss‚1.7419501888763547 4loss/FBt2/assignment_loss‚0.00017634064550809588 0loss/FRt1/detection_loss‚0.46077456468057143 %loss/total_loss‚8.141083043431124 0loss/FRt2/assignment_loss‚0.7863800663424302 2loss/SRqqt1/assignment_loss‚0.5735467893179381 /loss/FRt2/detection_loss‚1.5519575935541237 % _timestamp‚1.7845141619720407e+09 2loss/SRqqt2/assignment_loss‚0.5418589033560847 1loss/SRqqt2/detection_loss‚0.6190190817285469 0loss/FBt2/detection_loss‚0.43013447263898946 trainer/global_step‚8099 _step‚667 2loss/FBt1/assignment_loss‚0.005213656588454532 1loss/SRqqt1/detection_loss‚0.7498701162407031 /loss/FBt1/detection_loss‚0.6802012694614201 epoch‚20 _runtime‚7335.547394921›¾=ižå‚(:“ €öÒôÄÇË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚284.737' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.91233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðjžH`æ‚(:Õ €öÒ™¬¾Ì$ memory_percent‚95.49312591552734% disk.nvme0n1.out‚43223.9931640625 disk.md0.out‚ 220.3359375 disk.rbd0.in‚0" proc.memory.rssMB‚ 2947.7734375) proc.memory.percent‚17.99178123474121) disk./.usagePercent‚75.81220961339193# disk.dm-1.out‚11229.13623046875% proc.memory.availableMB‚ 738.40625 cpu‚25.686855750543366 proc.cpu.threads‚17 network.recv‚ 1.978028e+06% disk./.usageGB‚1163.9069480895996 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚23657.130859375 disk.md0.in‚0# disk.dm-0.out‚55180.90478515625 disk.rbd0.out‚0 network.sent‚ 1.2931016e+07 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚23657.130859375 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚43223.9931640625–šTkeç‚(:Ú öÒÃÎçË disk.nvme1n1.in‚0 disk.md0.in‚0) proc.memory.percent‚17.99178123474121 cpu‚25.687429335608513 proc.cpu.threads‚17 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚95.49322128295898 network.sent‚ 1.2971976e+07% disk./.usageGB‚1163.9069480895996 disk.md0.out‚ 220.64453125 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚43230.2978515625( disk.nvme1c1n1.out‚23657.67529296875& disk.nvme1n1.out‚23657.67529296875 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚55187.0048828125# disk.dm-1.out‚11229.18701171875 disk.rbd0.in‚0& proc.memory.availableMB‚ 738.390625" proc.memory.rssMB‚ 2947.7734375 network.recv‚ 1.982175e+06) disk./.usagePercent‚75.81220961339193% disk.nvme0n1.out‚43230.2978515625!Ô—ýœè‚(:‘ öÒü‚©Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.993' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.99766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S†zã2é- _wandbruntime‚7379  _runtime‚7379Ц¶ÇêÁ % _timestamp‚1.7845142056250272e+09 _runtime‚7379.200628692 _step‚668 & lr-AdamW/pg1‚0.0004064404200000228 & lr-AdamW/pg2‚0.0004064404200000228 trainer/global_step‚8149ŽñCÌëÆ _runtime‚7379.200628692 _step‚668 & lr-AdamW/pg1‚0.0004064404200000228 & lr-AdamW/pg2‚0.0004064404200000228 trainer/global_step‚8149 % _timestamp‚1.7845142056250272e+09œÏÑ<2ì- _wandbruntime‚7379  _runtime‚7379•EÎ, íóO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19483020 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9450920735188 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9465201419776 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146598458 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 trainer/global_step‚8149 _step‚669 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1247319662072320 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80625445 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33702773 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14220033 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚7379.201729064 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9464859995648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845142056260955e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_by›$6˜Ítes.small_pool.freed‚ 9464859995648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1247319662072320 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80635588 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19482740 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚280517473584128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1249471543931044 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270999022984704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1256784505028608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9465201419776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1247319645032960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9450580173914 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65973013 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65973015 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚280517601659904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270999004372480 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65973015 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1256784863492096 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1256784505028608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65973013 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9518469211648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1240020623195856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146608603 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14220036 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80635588 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80625445 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1247319645032960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33703056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9518578675200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1256784505028608 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65973013 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9518469211648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1240020623195856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146608603 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14220036 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80635588 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80625445 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1247319645032960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33703056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9518578675200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270999022984704 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1256784505028608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9465201419776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1247319645032960 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9450580173914 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65973013 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65973015 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚280517601659904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270999004372480 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65973015»hHv2ï- _wandbruntime‚7379  _runtime‚7379H›N'ðýM T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1256933889629696 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80645258 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1247467919535104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9465970094592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚297 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚670 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1247467936574464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14221767 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280551769739776 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1247467936574464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146615907 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9519585097216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33707076 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1240168036823968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9451688562166 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14221770 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1240168053863328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚7379.825284489 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9466311525888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1256934248100352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146615907 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33706776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9465970094592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146626066 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65980808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9452029123552 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80635101 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19485009 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ð‹ÙÝñ‚N OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚297 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚670 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1247467936574464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14221767 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280551769739776 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1247467936574464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146615907 E?D õî+eviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9519585097216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33707076 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1240168036823968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9451688562166 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14221770 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1240168053863328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚7379.825284489 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9466311525888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1256934248100352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146615907 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33706776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9465970094592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146626066 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65980808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9452029123552 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80635101 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19485009 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1256933889629696 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80645258 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1247467919535104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9465970094592žD@»2ò- _wandbruntime‚7379  _runtime‚7379{Bá½ó€ 2loss/SRqqt1/assignment_loss‚0.4786526803429455 0loss/SRqqt2/assignment_loss‚0.54816732574963 /loss/FBt2/detection_loss‚0.5615306580669166 _runtime‚ 7379.82586457 3loss/FBt1/assignment_loss‚0.0034808408521621987 1loss/FBt2/assignment_loss‚0.01851063522617414 /loss/FRt1/detection_loss‚0.4624766132542956 0loss/FBt1/detection_loss‚0.44475318441888495 $loss/total_loss‚7.72790536706369 0loss/FRt2/assignment_loss‚0.6887626595201746 1loss/SRqqt1/detection_loss‚0.7410371128960674 1loss/SRqqt2/detection_loss‚0.6193073946777331 trainer/global_step‚8149 /loss/FRt2/detection_loss‚1.4298909775978697 epoch‚20 $ _timestamp‚1.784514206250581e+09 _step‚671 0loss/FRt1/assignment_loss‚1.7313352844608363¸ËæM ô… 0loss/FRt1/assignment_loss‚1.7313352844608363 2loss/SRqqt1/assignment_loss‚0.4786526803429455 0loss/SRqqt2/assignment_loss‚0.54816732574963 /loss/FBt2/detection_loss‚0.5615306580669166 _runtime‚ 7379.82586457 3loss/FBt1/assignment_loss‚0.0034808408521621987 1loss/FBt2/assignment_loss‚0.01851063522617414 /loss/FRt1/detection_loss‚0.4624766132542956 0loss/FBt1/detection_loss‚0.44475318441888495 $loss/total_loss‚7.72790536706369 0loss/FRt2/assignment_loss‚0.6887626595201746 1loss/SRqqt1/detection_loss‚0.7410371128960674 1loss/SRqqt2/detection_loss‚0.6193073946777331 trainer/global_step‚8149 /loss/FRt2/detection_loss‚1.4298909775978697 epoch‚20 $ _timestamp‚1.784514206250581e+09 _step‚671Ÿ¦}–eõ‚(:Ú žöÒ§’éË" proc.memory.rssMB‚ 2947.7734375 proc.cpu.threads‚17 network.recv‚ 1.984535e+06 disk.md0.in‚0 disk.md0.out‚ 221.00390625# disk.dm-0.out‚55208.30322265625$ disk.nvme0n1.out‚43251.841796875( disk.nvme1c1n1.out‚23658.39306640625) proc.memory.percent‚17.99178123474121) disk./.usagePercent‚75.85514492872892 disk.nvme1n1.in‚0& disk.nvme1n1.out‚23658.39306640625 cpu‚25.688333578236062 network.sent‚ 1.2986824e+07% disk./.usageGB‚1164.5661125183105 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0" disk.dm-1.out‚11229.3935546875 disk.nvme0n1.in‚ 0.00390625$ memory_percent‚95.49477100372314( proc.memory.availableMB‚ 738.13671875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0c0n1.out‚43251.841796875 disk.nvme1c1n1.in‚0`æ)žö‚(:“ žöÒ»‰ñË gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.802' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.26733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý¥»½e÷‚(:Ú ­öÒæüËË network.sent‚ 1.30382e+07 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚23718.52001953125 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚95.4937219619751% disk./.usageGB‚1166.9061813354492 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚45346.21142578125 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚57361.3974609375" proc.memory.rssMB‚ 2947.7734375) disk./.usagePercent‚76.00756758413608 disk.nvme1c1n1.in‚0 disk.md0.out‚ 221.56640625" disk.dm-1.out‚11229.5888671875( proc.memory.availableMB‚ 738.30859375) proc.memory.percent‚17.99178123474121 proc.cpu.threads‚17 network.recv‚ 1.989147e+06( disk.nvme0c0n1.out‚45346.21142578125& disk.nvme1n1.out‚23718.52001953125 cpu‚25.688966027377013ï=Ä’ø‚(:‡ ­öÒÆóÌË gpu.0.gpu‚13.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.285' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.095 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CZÄbù‚(:× ¼öÒáÊË disk.md0.out‚ 221.87890625" proc.memory.rssMB‚ 2947.7734375 proc.cpu.threads‚17 network.sent‚ 1.3043143e+07( disk./.usagePercent‚76.0075785169926 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚45347.98291015625( disk.nvme1c1n1.out‚23719.07861328125 disk.md0.in‚0$ proc.memory.availableMB‚738.5625 network.recv‚ 1.992587e+06 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚11229.6748046875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚23719.07861328125) proc.memory.percent‚17.99178123474121$ disk./.usageGB‚1166.906349182129 disk.nvme1n1.in‚0$ memory_percent‚95.49217224121094 cpu‚25.689695362566784" disk.dm-0.out‚57362.9619140625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚45347.98291015625j,Oœú‚(:‘ ¼öÒ†¬”Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.394' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.13133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì™2û- _wandbruntime‚7423  _runtime‚7423òD}ÇüÁ trainer/global_step‚8199 % _timestamp‚1.7845142494569104e+09 _runtime‚7423.032505358 _step‚672 & lr-AdamW/pg1‚0.0004041067810357691 & lr-AdamW/pg2‚0.0004041067810357691=DÅTÌýÆ & lr-AdamW/pg1‚0.0004041067810357691 & lr-AdamW/pg2‚0.0004041067810357691 trainer/global_step‚8199 % _timestamp‚1.7845142494569104e+09 _runtime‚7423.032505358 _step‚672 P‰­F2þ- _wandbruntime‚7423  _runtime‚7423lÙ ÷ø'ÿòO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1256898202917418 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147482905 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1264254157701120 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1254733370140160 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚282232355762176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272658017876480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66362665 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9506403948596 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66362665 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚8199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1264253799237632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66362663 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚282232483837952 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81110097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14306736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1247392156569456 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9574447349248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9506063387322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1254733370140160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9520429097472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9574337885696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66362663 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1254733387179520 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1254733387179520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1264254157701120 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272658036488704 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1247392139530096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14306733 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81120240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33903849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147472760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9520429097472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19597113 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81120240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552–„§˜ý'€÷O aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272658017876480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66362665 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9506403948596 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66362665 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚8199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1264253799237632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66362663 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚282232483837952 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81110097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14306736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1247392156569456 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9574447349248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9506063387322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1254733370140160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9520429097472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9574337885696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66362663 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1254733387179520 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1254733387179520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1264254157701120 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272658036488704 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1247392139530096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14306733 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81120240 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33903849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147472760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9520429097472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19597113 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81120240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1256898202917418 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147482905 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1264254157701120 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1254733370140160 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚282232355762176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0¡Pºè02- _wandbruntime‚7423  _runtime‚7423‚¹oÄþ‚ûM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1247539570197568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66370456 H@DeviceStatsMonitor.on_train_batch_Pç($end/inactive_split.all.current‚291 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚ 7423.65646258 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9575553253888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9521539196416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33907585 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81129910 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272691198146560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9575456380416 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1247539587236928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66370456 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9521880627712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14308470 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19599406 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1257047099573888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147500368 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81119753 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1264403183838720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚674 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147490209 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147490209 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 trainer/global_step‚8199 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9507512336960 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1264403542309376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9521539196416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33907876 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1254881661681664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81119753 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9507171775574 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1254881644642304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1254881644642304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚282266770012672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845142500807173e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9521880627712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147500368—_'ƒ€N OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9521539196416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33907585 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81129910 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272691198146560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9575456380416 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1247539587236928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66370456 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9521880627712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14308470 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19599406 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1257047099573888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147500368 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81119753 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1264403183838720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚674 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147490209 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147490209 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 trainer/global_step‚8199 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9507512336960 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1264403542309376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9521539196416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33907876 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1254881661681664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81119753 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9507171775574 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1254881644642304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1254881644642304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚282266770012672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845142500807173e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9521880627712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147500368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1247539570197568 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66370456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚ 7423.65646258 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9575553253888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0¢Ì#Cï2„- _wandbruntime‚7423  _runtime‚7423Û¢à… % _timestamp‚1.7845142500818095e+09 0loss/FRt2/assignment_loss‚0.7643377104321042 .loss/FRt2/detection_loss‚1.423867143079708 .loss/FBt1/detection_loss‚0.501638026869235 %loss/total_loss‚7.961226306182472 trainer/global_step‚8199 _runtime‚7423.657139401 0loss/FRt1/detection_loss‚0.49665464586924885 /loss/FBt2/detection_loss‚0.6046686089978042 epoch‚20 _step‚675 0loss/FRt1/assignment_loss‚1.7153194057307604 1loss/SRqqt2/assignment_loss‚0.547229538194193 1loss/FBt1/assignment_loss‚0.03263834846764612 2loss/FBt2/assignment_loss‚0.007179010529065056 1loss/SRqqt2/detection_loss‚0.6059056749160613 2loss/SRqqt1/assignment_loss‚0.5049676015860922 1loss/SRqqt1/detection_loss‚0.75682059151055384QÄp †† _runtime‚7423.657139401 0loss/FRt1/detection_loss‚0.49665464586924885 /loss/FBt2/detection_loss‚0.6046686089978042 epoch‚20 _step‚675 0loss/FRt1/assignment_loss‚1.7153194057307604 1loss/SRqqt2/assignment_loss‚0.547229538194193 1loss/FBt1/assignment_loss‚0.03263834846764612 2loss/FBt2/assignment_loss‚0.007179010529065056 1loss/SRqqt2/detection_loss‚0.6059056749160613 2loss/SRqqt1/assignment_loss‚0.5049676015860922 1loss/SRqqt1/detection_loss‚0.7568205915105538 % _timestamp‚1.7845142500818095e+09 0loss/FRt2/assignment_loss‚0.7643377104321042 .loss/FRt2/detection_loss‚1.423867143079708 .loss/FBt1/detection_loss‚0.501638026869235 %loss/total_loss‚7.961226306182472 trainer/global_step‚8199£m‚@‘`‡‚(:Õ ËöÒ©¡ÇË network.sent‚ 1.3059009e+07& disk.nvme0c0n1.out‚46002.087890625 disk.nvme1n1.in‚0& disk.nvme1n1.out‚23949.82373046875 disk.md0.in‚0 disk.md0.out‚ 222.21875 disk.dm-0.in‚ 0.00390625$ memory_percent‚95.48280239105225 network.recv‚ 1.99623e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚23949.82373046875( proc.memory.availableMB‚ 740.09765625" proc.memory.rssMB‚ 2947.7734375) proc.memory.percent‚17.99178123474121 cpu‚25.690455434476192( disk./.usagePercent‚76.0045521035293 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625# disk.dm-0.out‚58247.02880859375% disk./.usageGB‚1166.8598861694336" disk.dm-1.out‚11229.7177734375$ disk.nvme0n1.out‚46002.087890625 disk.nvme1c1n1.in‚0 proc.cpu.threads‚17àvžˆ‚(:“ ËöÒí»ÈË gpu.0.gpu‚59.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚258.762' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.25399999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t®Ç«\‰‚(:Ñ ÚöÒœÊÄË% disk./.usageGB‚1166.8595924377441 disk.md0.out‚ 222.640625# disk.dm-0.out‚58247.90771484375 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 496.06640625" proc.memory.rssMB‚ 2947.7734375) proc.memory.percent‚17.99178123474121 disk.rbd0.out‚0' disk.nvme0c0n1.out‚46003.2919921875 network.sent‚ 1.3108422e+07 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625% disk.nvme0n1.out‚46003.2919921875$ disk.nvme1c1n1.out‚ 23950.5234375" disk.nvme1n1.out‚ 23950.5234375 network.recv‚ 2.000565e+06 disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚96.97225093841553 cpu‚25.6910783179222 proc.cpu.threads‚17# disk.dm-1.out‚11229.82666015625 disk.rbd0.in‚0) disk./.usagePercent‚76.00453297103039Á4œŠ‚(:‘ ÚöÒÒŸÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.96' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.986666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05T›&]‹‚(:Ò éöÒã÷Ë disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme0c0n1.out‚ 46006.9609375( disk.nvme1c1n1.out‚23951.26708984375) disk./.usagePercent‚76.00455533369146" disk.dm-0.out‚58251.2197265625# disk.dm-1.out‚11229.87744140625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0& disk.nvme1n1.out‚23951.26708984375 disk.md0.in‚0 disk.md0.out‚ 223.1328125 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0$ memory_percent‚96.92680835723877( proc.memory.availableMB‚ 503.51171875) proc.memory.percent‚18.06621551513672 cpu‚25.691297740567066 network.recv‚ 2.004311e+06 proc.memory.rssMB‚ 2959.96875 proc.cpu.threads‚17" disk.nvme0n1.out‚ 46006.9609375 network.sent‚ 1.3115114e+07$ disk./.usageGB‚1166.859935760498 disk.dm-1.in‚0 disk.rbd0.out‚0Ö©Ú“Œ‚(:ˆ éöÒ¹†¼Ë gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚241.893' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚80.631 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>éþ2- _wandbruntime‚7467  _runtime‚7467M*²ÉŽÃ ' lr-AdamW/pg2‚0.00040176692080175463 trainer/global_step‚8249 % _timestamp‚1.7845142935061803e+09 _runtime‚7467.081774682 _step‚676 ' lr-AdamW/pg1‚0.00040176692080175463VM%:ÎÈ ' lr-AdamW/pg1‚0.00040176692080175463 ' lr-AdamW/pg2‚0.00040176692080175463 trainer/global_step‚8249 % _timestamp‚1.7845142935061803e+09 _runtime‚7467.081774682 _step‚676¤Á§2- _wandbruntime‚7467  _runtime‚7467F‚Kca‘òO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1271723093446656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9630200873472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9561546600730 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81604892 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81594749 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 7467.08606614 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9576339623424 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1254763672903696 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1262147095247360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 trainer/global_step‚8249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283947360329728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9575998199296 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274317049992704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66752315 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66752313 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1271723451910144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274317031380480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19711778 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148357207 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1264325219504426 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1271723093446656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81604892 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1264325577105060 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 % _timestamp‚1.7845142935071847e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1254763689943056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14393433 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1271723451910144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9575998199296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66752313 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _step‚677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34105509 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1262147095247360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19712073 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66752315 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1262147112286720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14393436 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9630310337024 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81604892 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81594749 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 7467.08606614 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9576339623424 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1254763672903696 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1262147095247360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 trainer/global_step‚8249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283947360329728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9575998199296 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274317049992704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66752315 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66752313 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1271723451910144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274317031380480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19711778 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148357207 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1264325219504426 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1271723093446656 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81604892 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1264325577105060 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 % _timestamp‚1.7845142935071847e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1254763689943056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14393433 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1271723451910144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9575998199296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66752313 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _step‚677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34105509 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1262147095247360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19712073 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66752315 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1262147112286720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14393436 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9630310337024 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1254911103571168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19714295 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81604405 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1271872478047744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66760106 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66760108 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1262295369749504 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1262295369749504 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1264473758560150 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34109169 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9631424617472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9577108298240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66760106 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9577449729536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274350211650560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9631321452544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784514294130904e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34109465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1271872836518400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1262295386788864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274350230262784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283981654880256 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81614562 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148364511 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9577108298240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148374670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81614562 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1271872836518400 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148364511 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1262295386788864 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1254911103571168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19714295 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81604405 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1271872478047744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66760106 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66760108 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1262295369749504 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1262295369749504 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1264473758560150 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34109169 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9631424617472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9577108298240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66760106 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9577449729536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274350211650560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9631321452544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784514294130904e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34109465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1271872836518400 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1262295386788864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274350230262784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283981654880256 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81614562 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148364511 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9577108298240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148374670 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81614562 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1271872836518400 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148364511 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1262295386788864 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1279192387655680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1262135223316656 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67141965 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19826245 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275976063496704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9631908725248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9631908725248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚8299 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1279192746119168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67141963 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14480133 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9631567301120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34306066 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67141965 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9631567301120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303ÄRl‘þ'¤øO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1269560837393920 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1269560820354560 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1271752593692068 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9686153551360 _step‚681 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1279192387655680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1262135223316656 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67141965 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19826245 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275976063496704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9631908725248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9631908725248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚8299 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1279192746119168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67141963 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14480133 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9631567301120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34306066 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67141965 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9631567301120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9617370375412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1269560837393920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1269560820354560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14480136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 % _timestamp‚1.7845143369619086e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19825933 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1262135206277296 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82079401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9617029814138 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚275976044884480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1279342130727424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845143378930585e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9618138202390 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14481867 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1279341772256768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1269709094856704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9633018831360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82099214 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34310308 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1279342130727424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1271901132747904 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚Ôr7Ð 149248972 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34310022 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67149758 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19828438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9632677400064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9687268008448 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚8299 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7511.468835361 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9633018831360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82099214 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1262282653984128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67149756 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67149758 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1271900775147158 KÈ'§‚N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149238813 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9687158552064 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7511.468835361 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9633018831360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82099214 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1262282653984128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67149756 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67149758 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1271900775147158 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149248972 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67149756 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14481870 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276009225154560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1269709111896064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚285696383706624 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1262282636944768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚682 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149238813 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19828155 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1279342130727424 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845143378930585e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9618138202390 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14481867 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1279341772256768 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1269709094856704 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9633018831360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82099214 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34310308 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1279342130727424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1271901132747904 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149248972 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34310022 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67149758 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19828438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9632677400064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9687268008448 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚8299 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1276974545461760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚305 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1279179610279076 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34506736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277635077000704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1269506739650896 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1286661681864704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150105811 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1276974562501120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9687477827072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19939903 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚8349 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1286662040328192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34507041 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67531615 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303í‹,¥{ ¶÷O @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _step‚685 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82564053 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277635058388480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 7554.14383322 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9687136402944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1276974545461760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚305 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1279179610279076 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34506736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277635077000704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1269506739650896 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1286661681864704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150105811 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1276974562501120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9687477827072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19939903 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚8349 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1286662040328192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34507041 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67531615 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287377073608192 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1286811066465792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34510736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19942169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _step‚686 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1277122819963904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150113115 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9688587933184 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1286811424936448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82573709 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1277122837003264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1279327791734166 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1277122819963904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19942450 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150113115 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82583866 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82583866 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82573709 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67539408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9688246501888 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1277122837003264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9743112072192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9673961977184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9688246501888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚287411369342976 trainer/global_step‚8349 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1286811424936448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67539406 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1279328149334912 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1269654187357728 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150123274 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1286811424936448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67539406 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1279328149334912 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1269654187357728 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150123274 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1286811066465792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34510736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19942169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _step‚686 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1277122819963904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150113115 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9688587933184 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1286811424936448 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82573709 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1277122837003264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1279327791734166 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1277122819963904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19942450 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150113115 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82583866 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82583866 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82573709 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0®¤ïKæ2º- _wandbruntime‚7554  _runtime‚7554MY‡Ý »† 1loss/SRqqt1/assignment_loss‚0.544074379009209 3loss/FBt2/assignment_loss‚0.0076375041478532624 /loss/FRt1/detection_loss‚0.4782494511708513 0loss/FBt2/detection_loss‚0.47192111576857443 trainer/global_step‚8349 $ _timestamp‚1.784514381190594e+09 0loss/FRt1/assignment_loss‚1.6317279665682256 epoch‚20 /loss/FRt2/detection_loss‚1.5281243662340749 1loss/SRqqt1/detection_loss‚0.6304357204147998 1loss/SRqqt2/detection_loss‚0.6736941824646443 0loss/FBt1/detection_loss‚0.38857557861104847 %loss/total_loss‚7.496238707959144 _runtime‚7554.765881985 _step‚687 0loss/FRt2/assignment_loss‚0.6734967322326174 2loss/SRqqt2/assignment_loss‚0.4675726170332953 3loss/FBt1/assignment_loss‚0.0007290943039495215IKþˆ¼‹ /loss/FRt2/detection_loss‚1.5281243662340749 1loss/SRqqt1/detection_loss‚0.6304357204147998 1loss/SRqqt2/detection_loss‚0.6736941824646443 0loss/FBt1/detection_loss‚0.38857557861104847 %loss/total_loss‚7.496238707959144 _runtime‚7554.765881985 _step‚687 0loss/FRt2/assignment_loss‚0.6734967322326174 2loss/SRqqt2/assignment_loss‚0.4675726170332953 3loss/FBt1/assignment_loss‚0.0007290943039495215 1loss/SRqqt1/assignment_loss‚0.544074379009209 3loss/FBt2/assignment_loss‚0.0076375041478532624 /loss/FRt1/detection_loss‚0.4782494511708513 0loss/FBt2/detection_loss‚0.47192111576857443 trainer/global_step‚8349 $ _timestamp‚1.784514381190594e+09 0loss/FRt1/assignment_loss‚1.6317279665682256 epoch‚20¯"ª)¶f½‚(:Û ÒŽöÒ°ŠÊË% disk./.usageGB‚1166.6239547729492 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 227.27734375! disk.dm-1.out‚11230.630859375 disk.rbd0.in‚0) proc.memory.percent‚18.41263771057129 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚23959.32177734375& disk.nvme1n1.out‚23959.32177734375 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.98918448284965' disk.nvme0c0n1.out‚46023.3134765625 disk.rbd0.out‚0( proc.memory.availableMB‚ 532.62109375 cpu‚25.687886514616753 network.sent‚ 1.3275066e+07 network.recv‚ 2.030048e+06% disk.nvme0n1.out‚46023.3134765625 disk.nvme1c1n1.in‚0# disk.dm-0.out‚58266.06005859375$ memory_percent‚96.74913883209229" proc.memory.rssMB‚ 3016.72656252;êxž¾‚(:“ ÒŽöÒÈíÌ gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚262.132' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.37733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žý¨b¿‚(:× áŽöÒ´àË disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚23960.16259765625 disk.nvme1n1.in‚0 cpu‚25.68741523625096 proc.cpu.threads‚17& disk.nvme0n1.out‚46028.53662109375% disk./.usageGB‚1166.6239395141602 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625) proc.memory.percent‚18.41263771057129( disk./.usagePercent‚75.9891834889536 disk.nvme0c0n1.in‚ 0.00390625 disk.md0.in‚0 disk.md0.out‚ 227.6171875 network.recv‚ 2.034294e+06 disk.dm-1.in‚0" disk.dm-1.out‚11230.6787109375 disk.rbd0.in‚0& disk.nvme1n1.out‚23960.16259765625$ memory_percent‚96.74768447875977& proc.memory.availableMB‚ 532.859375" proc.memory.rssMB‚ 3016.7265625 network.sent‚ 1.3326529e+07! disk.dm-0.out‚58271.333984375( disk.nvme0c0n1.out‚46028.53662109375^¯ŒižÀ‚(:“ áŽöÒªè¦Ì gpu.0.gpu‚95.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚174.331' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚58.11033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U»›šÁ‚(:Ú ðŽöÒÖ·§Ë& disk.nvme1n1.out‚23960.57568359375 disk.md0.out‚ 227.8671875 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625' proc.memory.availableMB‚ 532.6015625" proc.memory.rssMB‚ 3016.7265625) proc.memory.percent‚18.41263771057129 network.recv‚ 2.037255e+06% disk./.usageGB‚1166.3861656188965 disk.md0.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚46029.50830078125 disk.nvme1c1n1.in‚0 cpu‚25.686917316515064 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625! disk.dm-0.out‚58272.111328125" disk.dm-1.out‚11230.7255859375 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0nã)ÏÃË1.out‚46029.50830078125( disk.nvme1c1n1.out‚23960.57568359375$ memory_percent‚96.74925804138184 network.sent‚ 1.3329451e+07) disk./.usagePercent‚75.97369585532626 disk.nvme1n1.in‚0îJó&žÂ‚(:“ ðŽöÒáÞÌ gpu.0.gpu‚47.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.845' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.28166666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éê72Ã- _wandbruntime‚7597  _runtime‚7597ªØëÉÄà _runtime‚7597.668808344 _step‚688 ' lr-AdamW/pg1‚0.00039471151726961317 ' lr-AdamW/pg2‚0.00039471151726961317 trainer/global_step‚8399 % _timestamp‚1.7845144240932288e+09`}íÞÎÅÈ ' lr-AdamW/pg2‚0.00039471151726961317 trainer/global_step‚8399 % _timestamp‚1.7845144240932288e+09 _runtime‚7597.668808344 _step‚688 ' lr-AdamW/pg1‚0.00039471151726961317°usœÇ2Æ- _wandbruntime‚7597  _runtime‚7597Ü{éÊø'ÇòO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1284388287608320 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150980113 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14653536 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34707864 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14653533 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚7597.669698435 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1294130976073728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1286606269265450 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9727996240954 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1284388287608320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67921263 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67921265 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83058848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83048705 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1284388270568960 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1284388270568960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83048705 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9797740224000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1294130976073728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚289091935997952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67921265 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1294131334537216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9728336802228 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150969968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20054046 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚289091812116480 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1294130976073728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚289091935997952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67921265 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1294131334537216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9728336802228 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150969968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20054046 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚289091812116480 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1294130976073728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1286606269265450 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9727996240954± ²12É- _wandbruntime‚7598  _runtime‚7598#éYê'ÊüM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34711862 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20056299 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9798857367040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1284536562110464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14655267 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9729445190592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1294280360674816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150987417 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1294280719145472 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1284536562110464 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1277025703691968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9744157035008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚289126109529600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1286755165921920 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67929056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9744157035008 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83058361 trainer/global_step‚8399 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9798962629120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _step‚690 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83068518 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1277025720731328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34711566 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150997576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20056592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1286754808321174 $ _timestamp‚1.784514424715335e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150997576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9729104629206 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1294280719145472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150987417 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83068518 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚289126233403904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279327270774784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67929058 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67929056 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83058361 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279327252162560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67929058 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1284536545071104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304Ä´åñ&ËN RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1284536545071104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83058361 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batc>ß@á"h_end/inactive_split_bytes.large_pool.freed‚279327252162560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67929058 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1284536545071104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34711862 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20056299 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9798857367040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1284536562110464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14655267 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9729445190592 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1294280360674816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150987417 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1294280719145472 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1284536562110464 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1277025703691968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9744157035008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚289126109529600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1286755165921920 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67929056 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9744157035008 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83058361 trainer/global_step‚8399 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9798962629120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _step‚690 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83068518 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1277025720731328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34711566 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150997576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20056592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1286754808321174 $ _timestamp‚1.784514424715335e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150997576 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9729104629206 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1294280719145472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150987417 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83068518 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚289126233403904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279327270774784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67929058 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67929056 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7393802404403687 particle/accuracy_2_of_4‚1 7validation_average_jet_accuracy‚0.30456623435020447 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6745842695236206 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2156045287847519 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4933556020259857 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4750553369522095 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3337100148200989 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.07213928550481796 /particle/accuracy_4_of_4‚0.9619753956794739 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2857142984867096 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1995970904827118 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9053126573562622 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.515884280204773 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18337500095367432 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.28025907278060913 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.27967673540115356 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45415836572647095 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_2‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 epoch‚20 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22996482253074646 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8485026359558105 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.28886687755584717 /particle/accuracy_3_of_4‚0.9965986609458923 jet/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚1 trainer/global_step‚8399 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9956710338592529 *jet/accuracy_2_of_2‚0.1176072284579277 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2956884205341339 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.506546676158905 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.2502085566520691 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25065749883651733 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02587849460542202 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 _runtime‚ 7614.33128039 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.31508880853652954 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47148585319519043 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2922011911869049 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 particle/accuracy_1_of_3‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20739339292049408 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7377928495407104 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9886165857315063 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5012107491493225 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6561822295188904 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.12695761024951935 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03884353116154671 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8443856239318848 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21243230998516083 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6510416865348816 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9966942071914673 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4630689024925232 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.5107327699661255 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7845144407552748e+09 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15076425671577454 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27690941095352173 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5703295469284058 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.3189438283443451 _step‚692 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45415836572647095 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47049304842948914 jet/accuracy_5_of_5‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.667931318283081 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987531304359436 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.007130124606192112 )particle/sensitivity‚0.62542325258255 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6460490226745605 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7249810695648193 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 *jet/accuracy_1_of_4‚0.9616956114768982 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2181384414434433 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2972952425479889 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02571740373969078 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5757157206535339 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5325997471809387 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2940317988395691 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882621765136719 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6525192260742188 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25119081139564514 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7519673705101013 particle/accuracy_2_of_3‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.25 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4572131633758545 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5533178448677063 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.490960031747818 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5909897089004517 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9910410642623901 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1963708996772766 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5408332943916321 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3236459195613861 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5527161359786987 *jet/accuracy_2_of_4‚0.7206095457077026 particle/accuracy_2_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5527161359786987 particle/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4661204516887665 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7208148837089539 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 +particle/specificity‚0.9042667150497437 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7369425892829895 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4923197031021118 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23375806212425232 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960261583328247 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5388498306274414 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5625458359718323 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03844699263572693 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01998046413064003 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.494680255651474 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19647341966629028 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2778840661048889 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8871941566467285 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981237053871155 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18602102994918823 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974259734153748 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6991648077964783 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48912742733955383 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7694946527481079 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962109923362732 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4202110767364502 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416667819023132 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.05458849295973778 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2645404040813446 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5652658939361572 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1386421173810959 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.758659839630127 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17655465006828308 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5451481938362122 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.3189438283443451 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 +jet/accuracy_2_of_3‚0.41929879784584045 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7306340336799622 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5050356984138489Óð©¨Ó³{ _runtime‚ 7614.33128039 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.31508880853652954 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47148585319519043 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2922011911869049 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 particle/accuracy_1_of_3‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20739339292049408 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7377928495407104 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9886165857315063 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5012107491493225 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6561822295188904 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.12695761024951935 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03884353116154671 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8443856239318848 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21243230998516083 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6510416865348816 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9966942071914673 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4630689024925232 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.5107327699661255 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7845144407552748e+09 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15076425671577454 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27690941095352173 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5703295469284058 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.3189438283443451 _step‚692 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45415836572647095 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47049304842948914 jet/accuracy_5_of_5‚0 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.667931318283081 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987531304359436 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.007130124606192112 )particle/sensitivity‚0.62542325258255 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6460490226745605 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7249810695648193 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 *jet/accuracy_1_of_4‚0.9616956114768982 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2181384414434433 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2972952425479889 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02571740373969078 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5757157206535339 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5325997471809387 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2940317988395691 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882621765136719 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6525192260742188 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25119081139564514 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7519673705101013 par×ÕÅt*ticle/accuracy_2_of_3‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.25 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4572131633758545 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5533178448677063 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.490960031747818 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5909897089004517 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9910410642623901 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1963708996772766 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5408332943916321 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3236459195613861 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5527161359786987 *jet/accuracy_2_of_4‚0.7206095457077026 particle/accuracy_2_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5527161359786987 particle/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4661204516887665 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7208148837089539 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 +particle/specificity‚0.9042667150497437 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7369425892829895 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4923197031021118 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23375806212425232 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960261583328247 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5388498306274414 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5625458359718323 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03844699263572693 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01998046413064003 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.494680255651474 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19647341966629028 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2778840661048889 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8871941566467285 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981237053871155 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18602102994918823 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974259734153748 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6991648077964783 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48912742733955383 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7694946527481079 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962109923362732 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4202110767364502 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416667819023132 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.05458849295973778 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2645404040813446 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5652658939361572 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1386421173810959 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.758659839630127 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17655465006828308 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5451481938362122 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.3189438283443451 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 +jet/accuracy_2_of_3‚0.41929879784584045 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7306340336799622 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5050356984138489 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2684151232242584 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7453325986862183 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_4‚0.08938658237457275 particle/accuracy_1_of_1‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.30095335841178894 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6495076417922974 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.055615730583667755 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.46678557991981506 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2240801304578781 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 jet/accuracy_2_of_5‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9071161150932312 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.013559768907725811 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9062248468399048 'particle/f_score‚0.6331712007522583 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.12328468263149261 )loss/val_total_loss‚7.721703053389272 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972149133682251 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6908333897590637 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 particle/accuracy_2_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9929646253585815 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9954156279563904 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.472618967294693 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5721631050109863 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18337500095367432 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4878358244895935 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7393802404403687 particle/accuracy_2_of_4‚1 7validation_average_jet_accuracy‚0.30456623435020447 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6745842695236206 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2156045287847519 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4933556020259857 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4750553369522095 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3337100148200989 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.07213928550481796 /particle/accuracy_4_of_4‚0.9619753956794739 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2857142984867096 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1995970904827118 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9053126573562622 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.515884280204773 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18337500095367432 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.28025907278060913 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.27967673540115356 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45415836572647095 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_2‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 epoch‚20 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22996482253074646 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8485026359558105 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.28886687755584717 /particle/accuracy_3_of_4‚0.9965986609458923 jet/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚1 trainer/global_step‚8399 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9956710338592529 *jet/accuracy_2_of_2‚0.1176072284579277 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2956884205341339 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.506546676158905 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.2502085566520691 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25065749883651733 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02587849460542202 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625´×žÔ‚(:“ ŽöÒ¿îÐÌ gpu.0.gpu‚62.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.185' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.72833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oÊŸjdÕ‚(:Ù ŽöÒ¡…€Í& disk.nvme1n1.out‚23962.43994140625 disk.dm-0.in‚ 0.00390625 cpu‚25.69154021591195 proc.cpu.threads‚17 network.sent‚ 1.3409549e+07 network.recv‚ 2.045183e+06 disk.rbd0.in‚0 disk.md0.out‚ 228.66796875! disk.dm-1.out‚11231.052734375 disk.nvme0c0n1.in‚ 0.00390625$ disk.nvme0n1.out‚46103.900390625( disk.nvme1c1n1.out‚23962.43994140625 disk.md0.in‚0# disk.dm-0.out‚58346.31005859375$ memory_percent‚96.76768779754639( proc.memory.availableMB‚ 529.58203125$ disk./.usageGB‚1166.688190460205 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚46103.900390625 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 3019.39453125* proc.memory.percent‚18.428921699523926) disk./.usagePercent‚75.99336853673377 disk.nvme0n1.in‚ 0.00390625ö|ÎfÖ‚(:Û öÒŽèÝË" disk.dm-0.out‚58487.8935546875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 network.recv‚ 2.049528e+06& disk.nvme1c1n1.out‚23989.220703125 disk.md0.in‚0 disk.md0.out‚ 229.06640625 disk.dm-0.in‚ 0.00390625# disk.dm-1.out‚11231.17333984375$ memory_percent‚96.76775932312012' proc.memory.availableMB‚ 529.5703125) disk./.usagePercent‚75.98922622648362& disk.nvme0n1.out‚46219.69091796875 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 3019.39453125* proc.memory.percent‚18.428921699523926 network.sent‚ 1.3455655e+07 disk.dm-1.in‚0( disk.nvme0c0n1.out‚46219.69091796875 cpu‚25.691010230262897 proc.cpu.threads‚17% disk./.usageGB‚1166.6245956420898 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0$ disk.nvme1n1.out‚23989.220703125ùz$“ׂ(:ˆ öÒ›Ù£Ì gpu.0.gpu‚38.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚290.016' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.672 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„S¬ŸeØ‚(:Ú ¬öÒÒ ÐË disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚46250.11279296875 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 3019.39453125 network.recv‚ 2.052266e+06 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0$ memory_percent‚96.76470756530762* proc.memory.percent‚18.428921699523926 cpu‚25.690572015759013 proc.cpu.threads‚17 disk.md0.out‚ 229.3671875& disk.nvme0n1.out‚46250.11279296875' proc.memory.availableMB‚ 530.0703125 network.sent‚ 1.3459543e+07) disk./.usagePercent‚75.98926548537749% disk./.usageGB‚1166.6251983642578 disk.nvme1n1.in‚0" disk.dm-0.out‚58545.5107421875 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme1c1n1.out‚24017.115234375$ disk.nvme1n1.out‚24017.115234375 disk.md0.in‚0# disk.dm-1.out‚11231.22412109375¬AÕžÙ‚(:“ ¬öÒ›Ø‘Ì gpu.0.gpu‚98.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.738' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.57933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A%·2Ú- _wandbruntime‚7657  _runtime‚7657ëˆOŠÆÛÀ trainer/global_step‚8449 $ _timestamp‚1.784514483854716e+09 _runtime‚7657.430312113 _step‚693 & lr-AdamW/pg1‚0.0003923482786559278 & lr-AdamW/pg2‚0.0003923482786559278‹nº}ËÜÅ _runtime‚7657.430312113 _step‚693 & lr-AdamW/pg1‚0.0003923482786559278 & lr-AdamW/pg2‚0.0003923482786559278 trainer/global_step‚8449 $ _timestamp‚1.784514483854716e+09µºäÝ2Ý- _wandbruntime‚7657  _runtime‚7657×.Ðø'ÞòO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34947618 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152048572 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14744349 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20203548 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1293744356591104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9818592442368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83624994 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20203269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1293744339551744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290850031426048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280975682671104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1295974845709579 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9874367367168 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68433723 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1303563290457600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68433723 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1303562931994112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83624994 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14744352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 $ _timestamp‚1.784514483855823e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9818592442368 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1303562931994112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68433721 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9803783303259 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1293744339551744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34947900 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83614851 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 trainer/global_step‚8449 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 _runtime‚7657.434028456 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9804123864533 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚694 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1293744356591104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚290850155307520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9818933866496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83614851 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9818933866496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 237041516474K©ß÷O ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1293744339551744 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290850031426048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚280975682671104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1295974845709579 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9874367367168 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68433723 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1303563290457600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68433723 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1303562931994112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83624994 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14744352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 $ _timestamp‚1.784514483855823e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9818592442368 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1303562931994112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68433721 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9803783303259 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1293744339551744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34947900 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83614851 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 trainer/global_step‚8449 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 _runtime‚7657.434028456 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9804123864533 ?9DeviceStatsMonitor.on_tr¤RÙ²T ain_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚694 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1293744356591104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚290850155307520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9818933866496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83614851 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9818933866496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34947618 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152048572 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14744349 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20203548 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1293744356591104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9818592442368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83624994 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20203269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0¶Nô2à- _wandbruntime‚7658  _runtime‚7658DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83634664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9804891691511 _runtime‚ 7658.05277879 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152066021 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34951643 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9875586749440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1286318493073792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9875489875968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1296123742366049 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68441516 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152076180 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83634664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1303712675065856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9820043972608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68441516 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152066021 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1286318510113152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14746083 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1303712316595200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1293892614053888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14746086 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152076180 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281008844328960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68441514 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1293892631093248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496ïœý='â€N =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281008844328960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68441514 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1293892631093248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20205560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281008862941184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83634664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9804891691511 _runtime‚ 7658.05277879 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152066021 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34951643 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9875586749440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1286318493073792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9875489875968 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1296123742366049 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68441516 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152076180 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83634664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1303712675065856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9820043972608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68441516 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152066021 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1286318510113152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14746083 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1303712316595200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1293892614053888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14746086 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152076180 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1311032226203136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84109646 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1293542612819280 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚7700.765074716 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68823371 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1311032584666624 ]HDeviceStatsMonitor.VLOèBon_train_batch_start/allocated_bytes.large_pool.freed‚1301158064658944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84099503 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1311032584666624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9874161544192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282634677562880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68823373 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1301158081698304 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9874502968320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20317492 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35148821 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35148541 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68823373 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68823371 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1293542595779920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1293542595779920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1311032226203136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84109646 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1293542612819280 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚7700.765074716 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68823371 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1311032584666624 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1301158064658944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84099503 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1311032584666624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9874161544192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚282634677562880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68823373 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1301158081698304 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9874502968320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20317492 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35148821 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35148541 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528º“³ Î2ò- _wandbruntime‚7701  _runtime‚7701ôÍÖ 'óýM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14832783 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1303550758953057 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84109159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14832786 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1293690026447392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292599349704704 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68831164 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1311181610804224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9860374904919 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152940323 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1301306356200448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845145278094301e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152940323 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9931473259520 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152950482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68831166 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9860715466305 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35152490 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9875271643136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35152768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1311181969274880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68831166 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚7701.385215334 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1303550401352311 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84119316 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9931365900288 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1293690043486752 _step‚699 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152950482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282667857832960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292599223733248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282667876445184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157ÇÎ"Rjô‚N QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35152490 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9875271643136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35152768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1311181969274880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68831166 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚7701.385215334 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1303550401352311 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84119316 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9931365900288 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1293690043486752 _step‚699 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152950482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚282667857832960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292599223733248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚282667876445184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14832783 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1303550758953057 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84109159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14832786 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1293690026447392 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMoni–p´Éžtor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292599349704704 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68831164 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1311181610804224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9860374904919 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152940323 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1301306356200448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845145278094301e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152940323 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9931473259520 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152950482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68831166 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9860715466305»hrÊÔ2õ- _wandbruntime‚7701  _runtime‚7701ì/9ß ö† % _timestamp‚1.7845145278105206e+09 2loss/SRqqt1/assignment_loss‚0.6705345594605584 2loss/FBt1/assignment_loss‚0.037227586433520456 /loss/FBt1/detection_loss‚0.5605915233833608 /loss/FBt2/detection_loss‚0.5971665237572908 _runtime‚7701.385960954 0loss/FRt1/assignment_loss‚1.7732821571851214 0loss/FRt2/assignment_loss‚0.6477711410572131 2loss/SRqqt2/assignment_loss‚0.5365293628018446 0loss/SRqqt1/detection_loss‚0.848812318327772 1loss/SRqqt2/detection_loss‚0.6426004442941717 epoch‚21 trainer/global_step‚8499 _step‚700 /loss/FRt2/detection_loss‚1.3904264437946645 %loss/total_loss‚8.183645072059402 4loss/FBt2/assignment_loss‚0.00028258942156065913 0loss/FRt1/detection_loss‚0.478420422142324055ƒœJ÷‹ _runtime‚7701.385960954 0loss/FRt1/assignment_loss‚1.7732821571851214 0loss/FRt2/assignment_loss‚0.6477711410572131 2loss/SRqqt2/assignment_loss‚0.5365293628018446 0loss/SRqqt1/detection_loss‚0.848812318327772 1loss/SRqqt2/detection_loss‚0.6426004442941717 epoch‚21 trainer/global_step‚8499 _step‚700 /loss/FRt2/detection_loss‚1.3904264437946645 %loss/total_loss‚8.183645072059402 4loss/FBt2/assignment_loss‚0.00028258942156065913 0loss/FRt1/detection_loss‚0.47842042214232405 % _timestamp‚1.7845145278105206e+09 2loss/SRqqt1/assignment_loss‚0.6705345594605584 2loss/FBt1/assignment_loss‚0.037227586433520456 /loss/FBt1/detection_loss‚0.5605915233833608 /loss/FBt2/detection_loss‚0.5971665237572908¼NêÔÖiø‚(:Þ èöÒè‚‚Ì# proc.memory.rssMB‚ 3019.39453125 cpu‚25.688539292632495 proc.cpu.threads‚17) disk./.usagePercent‚76.00471733874713% disk./.usageGB‚1166.8624229431152 disk.md0.out‚ 230.73828125" disk.dm-1.out‚11231.6552734375( disk.nvme0c0n1.out‚46498.51806640625 disk.nvme0n1.in‚ 0.00390625& disk.nvme1n1.out‚24051.07763671875 network.recv‚ 2.066829e+06 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme1c1n1.out‚24051.07763671875 disk.nvme1n1.in‚0$ memory_percent‚96.76530361175537 disk.dm-0.in‚ 0.00390625! disk.dm-0.out‚58824.462890625 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚46498.51806640625( proc.memory.availableMB‚ 529.97265625* proc.memory.percent‚18.428921699523926 network.sent‚ 1.3547434e+07 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0{ÁVžù‚(:“ èöÒÎßÆÌ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚272.384' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.79466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YŠÑ»aú‚(:Ö ÷öÒü¡ÇË disk.nvme0c0n1.in‚ 0.00390625# proc.memory.rssMB‚ 3019.39453125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚46503.5302734375& disk.nvme1n1.out‚24051.95458984375 disk.md0.out‚ 231.14453125( proc.memory.availableMB‚ 529.97265625 cpu‚25.688047976892257 network.sent‚ 1.3599247e+07 network.recv‚ 2.071339e+06 disk.nvme1c1n1.in‚0# disk.dm-0.out‚58829.38818359375* proc.memory.percent‚18.428921699523926) disk./.usagePercent‚75.98924759524863% disk./.usageGB‚1166.6249237060547( disk.nvme1c1n1.out‚24051.95458984375 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚11231.75' disk.nvme0c0n1.out‚46503.5302734375$ memory_percent‚96.76530361175537 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 disk.rbd0.out‚0 JÊ¢Ÿû‚(:” ÷öÒéí‹Ì gpu.0.gpu‚77.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚127.779' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚42.592999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Íu?Y`ü‚(:Õ †öÒ¼³êË disk.nvme1n1.in‚0# disk.dm-0.out‚58830.25146484375 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0' proc.memory.availableMB‚ 529.6484375( disk./.usagePercent‚75.9892677216436 disk.dm-1.in‚0( disk.nvme1c1n1.out‚24052.66064453125 proc.cpu.threads‚17& disk.nvme1n1.out‚24052.66064453125 disk.md0.in‚0 disk.md0.out‚ 231.578125 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚46504.646484375$ disk.nvme0n1.out‚46504.646484375* proc.memory.percent‚18.428921699523926 network.sent‚ 1.3604161e+07 network.recv‚ 2.074555e+06% disk./.usageGB‚1166.6252326965332 disk.dm-1.out‚ 11231.7890625 disk.rbd0.out‚0$ memory_percent‚96.76728248596191# proc.memory.rssMB‚ 3019.39453125 cpu‚25.687521243835477óíóâ›ý‚(: †öÒ§€°Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.7' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.233333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-Á2þ- _wandbruntime‚7744  _runtime‚77443;¢ßÉÿà ' lr-AdamW/pg2‚0.00038760555543665437 trainer/global_step‚8549 % _timestamp‚1.7845145707265093e+09 _runtime‚7744.303068185 _step‚701 ' lr-AdamW/pg1‚0.00038760555543665437žÇ¿kÎ€È ' lr-AdamW/pg1‚0.00038760555543665437 ' lr-AdamW/pg2‚0.00038760555543665437 trainer/global_step‚8549 % _timestamp‚1.7845145707265093e+09 _runtime‚7744.303068185 _step‚701½åÔ é2- _wandbruntime‚7744  _runtime‚7744ßù'‚óO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84584155 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35349917 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1308571789766144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1308571806805504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20432168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚294279749303296 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9986167699968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1310829236484229 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69213021 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35350206 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1300914146192880 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153807321 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1318501520412160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9930072070144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9929730646016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9929730646016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1318501878875648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69213023 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9986058236416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 % _timestamp‚1.7845145707272632e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9930072070144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20432454 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284293709679104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚702 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84594298 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1308571806805504 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69213021 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚8549 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69213023’ÉEþ'ƒøO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20432454 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284293709679104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚702 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84594298 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1308571806805504 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69213021 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚8549 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69213023 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84584155 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35349917 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1308571789766144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1308571806805504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20432168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚294279749303296 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9986167699968 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1310829236484229 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69213021 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35350206 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1300914146192880 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153807321 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1318501520412160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9930072070144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9929730646016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9929730646016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1318501878875648 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69213023 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9986058236416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 % _timestamp‚1.7845145707272632e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9930072070144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85¾yM‹62„- _wandbruntime‚7744  _runtime‚7744<Ƀùo…üM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84603968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1318651263483904 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚703 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153814625 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9930840744960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1310977417939319 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1308720064268288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9931182176256 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9931182176256 XJDeviceStatsMoni2Zúœ“tor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153824784 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9930840744960 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1308720081307648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35354265 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1301061576860352 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69220816 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚294314044938240 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1308720064268288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20434779 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚294314173006848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1301061559820992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69220816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9915858118327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84603968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1318651263483904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69220814 $ _timestamp‚1.784514571349027e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153824784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153814625 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35353968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _runtime‚7744.924784516 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1318650905013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284326871336960 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9916198679713).„`'†N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153814625 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9930840744960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1310977417939319 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1308720064268288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9931182176256 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9931182176256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153824784 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9930840744960 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1308720081307648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35354265 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1301061576860352 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69220816 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚294314044938240 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1308720064268288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20434779 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚294314173006848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1301061559820992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69220816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9915858118327 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84603968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1318651263483904 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69220814 $ _timestamp‚1.784514571349027e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153824784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153814625 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35353968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _runtime‚7744.924784516 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1318650905013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284326871336960 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9916198679713 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84603968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1318651263483904 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚703¿ :|2‡- _wandbruntime‚7744  _runtime‚7744k´s ˆ‡ &loss/total_loss‚7.6072905140019245 epoch‚21 3loss/SRqqt2/assignment_loss‚0.45750497068489787 3loss/FBt2/assignment_loss‚8.778711406694612e-05 1loss/SRqqt1/detection_loss‚0.7613449322710709 1loss/SRqqt2/detection_loss‚0.5719675779019293 _step‚704 /loss/FRt1/assignment_loss‚1.655981823381397 /loss/FBt2/detection_loss‚0.4711069119499719 _runtime‚7744.925316737 0loss/FRt2/assignment_loss‚0.7270350023456241 2loss/SRqqt1/assignment_loss‚0.5562850630003878 2loss/FBt1/assignment_loss‚0.004974347336406027 0loss/FRt1/detection_loss‚0.47150009252675135 /loss/FRt2/detection_loss‚1.4129021138934743 /loss/FBt1/detection_loss‚0.5165998915959473 trainer/global_step‚8549 % _timestamp‚1.7845145713500464e+09Cƒš‰Œ 0loss/FRt2/assignment_loss‚0.7270350023456241 2loss/SRqqt1/assignment_loss‚0.5562850630003878 2loss/FBt1/assignment_loss‚0.004974347336406027 0loss/FRt1/detection_loss‚0.47150009252675135 /loss/FRt2/detection_loss‚1.4129021138934743 /loss/FBt1/detection_loss‚0.5165998915959473 trainer/global_step‚8549 % _timestamp‚1.7845145713500464e+09 &loss/total_loss‚7.6072905140019245 epoch‚21 3loss/SRqqt2/assignment_loss‚0.45750497068489787 3loss/FBt2/assignment_loss‚8.778711406694612e-05 1loss/SRqqt1/detection_loss‚0.7613449322710709 1loss/SRqqt2/detection_loss‚0.5719675779019293 _step‚704 /loss/FRt1/assignment_loss‚1.655981823381397 /loss/FBt2/detection_loss‚0.4711069119499719 _runtime‚7744.925316737Àªì9XŠ‚(:Í •öÒÿƒ³Ë( disk.nvme0c0n1.out‚46515.65673828125 disk.nvme0n1.in‚ 0.00390625# memory_percent‚96.7581033706665 network.recv‚ 2.07832e+06% disk./.usageGB‚1166.3878631591797 disk.nvme1n1.in‚0 disk.md0.out‚ 232.12890625 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚ 58841.7734375 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚46515.65673828125( disk./.usagePercent‚75.9738064262615" disk.nvme1c1n1.out‚ 24054.53125 disk.md0.in‚0( proc.memory.availableMB‚ 531.15234375# proc.memory.rssMB‚ 3019.39453125* proc.memory.percent‚18.428921699523926 cpu‚25.68703360944315 disk.nvme1n1.out‚ 24054.53125 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 1.3607125e+07 disk.nvme1c1n1.in‚0 disk.dm-1.in‚0# disk.dm-1.out‚11231.96044921875 disk.rbd0.out‚0fúiО‹‚(:“ •öÒ¥ŸžÌ gpu.0.gpu‚50.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.722' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.57399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨÷]Œ‚(:Ò ¤öÒÚÎÜË% disk./.usageGB‚1166.1490898132324 disk.nvme1c1n1.in‚0! disk.dm-0.out‚58845.310546875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚46519.49853515625 network.sent‚ 1.3671361e+07 network.recv‚ 2.082455e+06 disk.md0.out‚ 232.4921875 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚46519.49853515625 cpu‚25.686895798859776 proc.cpu.threads‚17$ disk.nvme1c1n1.out‚ 24055.0703125 disk.nvme1n1.in‚0" disk.nvme1n1.out‚ 24055.0703125 disk.dm-0.in‚ 0.00390625# memory_percent‚96.7580795288086* proc.memory.percent‚18.428921699523926 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625% proc.memory.availableMB‚ 531.15625# proc.memory.rssMB‚ 3019.39453125 disk.dm-1.in‚0# disk.dm-1.out‚11232.01513671875) disk./.usagePercent‚75.95825369244308y6ühž‚(:“ ¤öÒïöÌ gpu.0.gpu‚97.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚260.579' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.85966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N<^Ž‚(:Ó ³öÒÏÞ­Ë* proc.memory.percent‚18.428921699523926 network.sent‚ 1.3676211e+07 disk.rbd0.out‚0% proc.memory.availableMB‚ 531.15625 proc.cpu.threads‚17 disk.md0.out‚ 233.0703125% disk.nvme0c0n1.out‚46536.18359375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.dm-1.out‚11232.10498046875 disk.rbd0.in‚0# memory_percent‚96.7580795288086 cpu‚25.68727603450904 network.recv‚ 2.086199e+06% disk./.usageGB‚1166.2037658691406' disk.nvme1c1n1.out‚24057.8115234375% disk.nvme1n1.out‚24057.8115234375 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚58863.42236328125 disk.dm-1.in‚0# proc.memory.rssMB‚ 3019.39453125) disk./.usagePercent‚75.96181507045371 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625# disk.nvme0n1.out‚46536.18359375 disk.md0.in‚0]K-“‚(:ˆ ³öÒ„¸îË gpu.0.gpu‚92.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚234.642' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚78.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êr¹A2- _wandbruntime‚7786  _runtime‚7786>ÞÿdÇ‘Á % _timestamp‚1.7845146132050207e+09 _runtime‚7786.780584275 _step‚705 & lr-AdamW/pg1‚0.0003852263754473162 & lr-AdamW/pg2‚0.0003852263754473162 trainer/global_step‚8599q^ó Ì’Æ % _timestamp‚1.7845146132050207e+09 _runtime‚7786.780584275 _step‚705 & lr-AdamW/pg1‚0.0003852263754473162 & lr-AdamW/pg2‚0.0003852263754473162 trainer/global_step‚8599Áž 2“- _wandbruntime‚7786  _runtime‚7786¤ÈDÌ"”õO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295994725130240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15004449 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚285952723183104 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154671478 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1315985514873344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9985299747840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1315985531912704 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1318255895470603 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9985641171968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20545925 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9970232943483 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69602671 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85068807 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85078950 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9985299747840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154681623 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1325970814621184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35550090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚285952704570880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295994594957312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69602671 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0m®: (•úO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9985299747840 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1315985531912704 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1318255895470603 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9985641171968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20545925 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9970232943483 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69602671 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85068807 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85078950 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9985299747840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154681623 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1325970814621184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35550090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚285952704570880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295994594957312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69602671 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295994725130240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15004449 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚285952723183104 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154671478 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1315985514873344ÂÒ—ëÆ2–- _wandbruntime‚7787  _runtime‚7787ýœ¥ž'—ÿM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚285985884840960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85078463 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1316133789375488 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85078463 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20548139 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15006186 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69610466 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚296029016416768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1308433110233952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1316133806414848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69610464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1318404792127073 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _runtime‚7787.710605897 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9986409846784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10043112963584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10043005604352 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1318404434526327 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1326120557692928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9986409846784 _step‚707 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1326120557692928 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1316133789375488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154699086 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1316133806414848 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1326120199222272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35554059 % _timestamp‚1.7845146141341069e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85088620 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9971341331735 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154699086 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚296028890445312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8599ü ì1 '˜„N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9971341331735 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154699086 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚296028890445312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8599 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚285985884840960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85078463 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1316133789375488 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85078463 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20548139 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15006186 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69610466 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚296029016416768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1308433110233952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1316133806414848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69610464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1318404792127073 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _runtime‚7787.710605897 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9986409846784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10043112963584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10043005604352 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1318404434526327 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1326120557692928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9986409846784 _step‚707 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1326120557692928 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1316133789375488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154699086 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1316133806414848 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1326120199222272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35554059 % _timestamp‚1.7845146141341069e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85088620 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360Ã7;f}2™- _wandbruntime‚7787  _runtime‚7787mS¦? šƒ /loss/FRt1/assignment_loss‚1.715517945761608 2loss/SRqqt2/assignment_loss‚0.5254732562656785 /loss/FBt2/detection_loss‚0.4061845892907253 _runtime‚7787.711213948 2loss/SRqqt1/assignment_loss‚0.5473356231135161 2loss/FBt1/assignment_loss‚0.008279449705661152 /loss/FRt1/detection_loss‚0.5022724076540361 /loss/FRt2/detection_loss‚1.3832971593272128 0loss/SRqqt1/detection_loss‚0.904211716777612 % _timestamp‚1.7845146141352053e+09 2loss/FBt2/assignment_loss‚0.013419826427310762 1loss/SRqqt2/detection_loss‚0.6351150128960115 0loss/FBt1/detection_loss‚0.43862027784847873 %loss/total_loss‚7.810442446248192 epoch‚21 _step‚708 0loss/FRt2/assignment_loss‚0.7307151811803411 trainer/global_step‚8599Û›ˆ 0loss/SRqqt1/detection_loss‚0.904211716777612 % _timestamp‚1.7845146141352053e+09 2loss/FBt2/assignment_loss‚0.013419826427310762 1loss/SRqqt2/detection_loss‚0.6351150128960115 0loss/FBt1/detection_loss‚0.43862027784847873 %loss/total_loss‚7.8104424462481');ñ92 epoch‚21 _step‚708 0loss/FRt2/assignment_loss‚0.7307151811803411 trainer/global_step‚8599 /loss/FRt1/assignment_loss‚1.715517945761608 2loss/SRqqt2/assignment_loss‚0.5254732562656785 /loss/FBt2/detection_loss‚0.4061845892907253 _runtime‚7787.711213948 2loss/SRqqt1/assignment_loss‚0.5473356231135161 2loss/FBt1/assignment_loss‚0.008279449705661152 /loss/FRt1/detection_loss‚0.5022724076540361 /loss/FRt2/detection_loss‚1.3832971593272128ÄÉÒ˜@cœ‚(:Ø ÂöÒú¦¹Ë disk.nvme1c1n1.in‚0 network.recv‚ 2.089523e+06% disk./.usageGB‚1166.7138214111328 disk.rbd0.out‚0& disk.nvme0c0n1.out‚46666.376953125$ disk.nvme0n1.out‚46666.376953125 disk.md0.in‚0* proc.memory.percent‚18.428921699523926 disk.nvme0n1.in‚ 0.00390625 disk.md0.out‚ 233.796875 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0( proc.memory.availableMB‚ 531.08984375 cpu‚25.68797151679962 proc.cpu.threads‚17 disk.nvme1n1.in‚0% disk.nvme1n1.out‚24126.4619140625 network.sent‚ 1.3679133e+07) disk./.usagePercent‚75.99503803361881 disk.nvme0c0n1.in‚ 0.00390625# disk.dm-0.out‚59060.54345703125# disk.dm-1.out‚11232.30810546875$ memory_percent‚96.75848484039307 disk.rbd0.in‚0' disk.nvme1c1n1.out‚24126.4619140625# proc.memory.rssMB‚ 3019.394531252ó}ž‚(:“ ÂöÒ‹é©Ì gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚270.643' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.21433333333331 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò.ýfž‚(:Û ÑöÒá½±Ë disk.md0.out‚ 234.24609375% disk.nvme0n1.out‚46794.4736328125& disk.nvme1n1.out‚24180.28369140625# proc.memory.rssMB‚ 3019.39453125* proc.memory.percent‚18.428921699523926 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚46794.4736328125 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 531.0859375 cpu‚25.68863343886425% disk./.usageGB‚1166.6245498657227 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0! disk.dm-1.out‚11232.408203125 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚24180.28369140625 network.sent‚ 1.3743507e+07 network.recv‚ 2.093985e+06) disk./.usagePercent‚75.98922324479548# disk.dm-0.out‚59241.39501953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0$ memory_percent‚96.75850868225098t^„¾žŸ‚(:“ ÑöÒáþëË gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚267.756' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.25199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ïiž ‚(:“ àöÒ‡ìßË gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚268.737' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.57900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xàd¡‚(:Ù àöÒèߪÌ network.sent‚ 1.3748324e+07 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625& disk.nvme1n1.out‚24774.08154296875 disk.md0.in‚0# proc.memory.rssMB‚ 3019.39453125 proc.cpu.threads‚17 network.recv‚ 2.097228e+06% disk./.usageGB‚1166.6245346069336 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0c0n1.out‚46849.986328125( disk.nvme1c1n1.out‚24774.08154296875# disk.dm-0.out‚59355.41845703125$ memory_percent‚96.75848484039307 cpu‚25.689317766922745) disk./.usagePercent‚75.98922225089943 disk.rbd0.out‚0$ disk.nvme0n1.out‚46849.986328125 disk.md0.out‚ 234.453125 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0! disk.dm-1.out‚11767.248046875( proc.memory.availableMB‚ 531.08984375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0* proc.memory.percent‚18.428921699523926Xv7ã2¢- _wandbruntime‚7831  _runtime‚7831_0<Ç£Á trainer/global_step‚8649 % _timestamp‚1.7845146576110685e+09 _runtime‚7831.186618509 _step‚709 & lr-AdamW/pg1‚0.0003828421868413872 & lr-AdamW/pg2‚0.0003828421868413872¥ùêáÌ¤Æ _runtime‚7831.186618509 _step‚709 & lr-AdamW/pg1‚0.0003828421868413872 & lr-AdamW/pg2‚0.0003828421868413872 trainer/global_step‚8649 % _timestamp‚1.7845146576110685e+09Å£·]ù2¥- _wandbruntime‚7831  _runtime‚78317©ÿ'¦ùO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15091149 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1325683269658245 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15091152 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69992321 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10097717095936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1333440108830208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287611718074880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297709550663680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 155545780 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85553459 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10040868849664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 $ _timestamp‚1.784514657611896e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚710 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35751383 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1333440108830208 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10041210273792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10026056718165 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1323399239980544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155555925 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20660510 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1315657212940080 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1333440467293696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1323399239980544 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1333440467293696 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1323399257019904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85563602 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1315657195900720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85553459 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1323399257019904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20660234 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69992321 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69992323 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 trainer/global_step‚8649 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10041210273792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69992323 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚7831.187389851 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287611736687104 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1325682912057611 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10025716156891 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10040868849664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35751662 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0tÌêè(§þO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69992323 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 trainer/global_step‚8649 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10041210273792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69992323 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚7831.187389851 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287611736687104 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1325682912057611 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10025716156891 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10040868849664 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35751662 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15091149 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1325683269658245 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15091152 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69992321 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10097717095936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1333440108830208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287611718074880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297709550663680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 155545780 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85553459 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10040868849664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 $ _timestamp‚1.784514657611896e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚710 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35751383 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1333440108830208 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10041210273792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10026056718165 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1323399239980544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155555925 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20660510 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1315657212940080 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1333440467293696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1323399239980544 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1333440467293696 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1323399257019904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85563602 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1315657195900720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85553459 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1323399257019904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20660234 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69992321Æ!C‡2¨- _wandbruntime‚7831  _runtime‚7831BL•s©…N SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10041978948608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297743850199552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10042320379904 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1315804643607552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70000116 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10098825883136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10042320379904 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1323547531522048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155573388 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15092883 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1315804626568192 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1333589851901952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70000114 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1333589493431296 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287644898344960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155563229 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85563115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1323547514482688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20662780 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85573272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8649 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1333589851901952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155563229 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297743724228096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DϪИ eviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35755666 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85563115 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1325831808714081 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1323547514482688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15092886 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35755372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10026824545143 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚7831.812045679 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70000114 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1325831451113335ŠÁÍ 'ªŠN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155573388 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1315804643607552 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70000116 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10098825883136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10042320379904 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1323547531522048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155573388 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15092883 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1315804626568192 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1333589851901952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70000114 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1333589493431296 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287644898344960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155563229 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85563115 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1323547514482688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20662780 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85573272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8649 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1333589851901952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155563229 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297743724228096 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35755666 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85563115 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1325831808714081 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1323547514482688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15092886 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35755372 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10026824545143 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚7831.812045679 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70000114 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1325831451113335 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10041978948608 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297743850199552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10042320379904 F´G^°‚(:Ó þöÒŽå«Ë$ disk.nvme0c0n1.out‚ 46867.0859375 disk.nvme0n1.in‚ 0.00390625 disk.md0.out‚ 235.38671875 disk.rbd0.in‚0* proc.memory.percent‚18.428921699523926% disk./.usageGB‚1166.6250228881836( disk.nvme1c1n1.out‚24775.81591796875& disk.nvme1n1.out‚24775.81591796875 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.dm-0.out‚ 59371.9921875 disk.dm-1.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚75.98925405557294" disk.nvme0n1.out‚ 46867.0859375 disk.nvme1n1.in‚0# disk.dm-1.out‚11767.50341796875$ memory_percent‚96.76337242126465 cpu‚25.69066663780025 network.sent‚ 1.3814826e+07 disk.nvme1c1n1.in‚0' proc.memory.availableMB‚ 530.2890625# proc.memory.rssMB‚ 3019.39453125 proc.cpu.threads‚17 network.recv‚ 2.104795e+06 disk.nvme0c0n1.in‚ 0.00390625ÙFÛäž±‚(:“ þöÒÍ‹ŠÌ gpu.0.gpu‚94.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚269.423' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.80766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!:ôøh²‚(:Ý ‘öÒ̇°Ë disk.nvme1n1.in‚0# proc.memory.rssMB‚ 3019.39453125 cpu‚25.69138112309501 proc.cpu.threads‚17% disk./.usageGB‚1166.6255798339844 disk.dm-1.in‚0( disk.nvme0c0n1.out‚46876.84130859375 disk.md0.in‚0 disk.md0.out‚ 235.77734375) disk./.usagePercent‚75.98929033277867 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625% disk.nvme1n1.out‚24777.1962890625( proc.memory.availableMB‚ 529.91796875 network.recv‚ 2.108282e+06" disk.dm-0.out‚59382.1591796875# disk.dm-1.out‚11767.62353515625 disk.rbd0.in‚0& disk.nvme0n1.out‚46876.84130859375 disk.nvme1c1n1.in‚0$ memory_percent‚96.76563739776611* proc.memory.percent‚18.428921699523926 network.sent‚ 1.3820479e+07 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0' disk.nvme1c1n1.out‚24777.1962890625øþB•ž³‚(:“ ‘öÒîô±Ë gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚270.716' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.23866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02ª¯Ñ2´- _wandbruntime‚7875  _runtime‚78758~Ê÷ǵÁ % _timestamp‚1.7845147019960768e+09 _runtime‚7875.571677249 _step‚713 & lr-AdamW/pg1‚0.0003804531427508464 & lr-AdamW/pg2‚0.0003804531427508464 trainer/global_step‚8699…ëôÇÌ¶Æ _runtime‚7875.571677249 _step‚713 & lr-AdamW/pg1‚0.0003804531427508464 & lr-AdamW/pg2‚0.0003804531427508464 trainer/global_step‚8699 % _timestamp‚1.7845147019960768e+09ÉFÝ6›2·- _wandbruntime‚7875  _runtime‚78759A!…ÿ'¸ùO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1330812965087744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156430227 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86048254 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10153510202880 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚289270750191104 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86048254 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚ 7875.57249559 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70381971 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35952110 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1340909403039232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70381973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15177849 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 115755008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _step‚714 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1330812982127104 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1340909761502720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20774261 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10096437951488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15177852 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299424241781760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10096437951488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86038111 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1323028746313680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 134367232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1330812982127104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1340909403039232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚289270731578880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70381973 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10096779375616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10153625957888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1333110286245253 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70381971 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20774553 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156420082 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8699 $ _timestamp‚1.784514701996859e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1340909761502720 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1323028729274320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚299424376148992 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1333109928644619 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86038111 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35952405 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1330812965087744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35952110 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1340909403039232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70381973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15177849 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 115755008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _step‚714 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1330812982127104 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1340909761502720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20774261 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10096437951488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15177852 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299424241781760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10096437951488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86038111 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1323028746313680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 134367232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚³oµùn0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1330812982127104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1340909403039232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚289270731578880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70381973 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10096779375616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10153625957888 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1333110286245253 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70381971 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20774553 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156420082 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8699 $ _timestamp‚1.784514701996859e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1340909761502720 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1323028729274320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚299424376148992 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1333109928644619 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86038111 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35952405 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1330812965087744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1330812965087744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156430227 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86048254 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10153510202880 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚289270750191104 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86048254 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180Ê´5ÚÓ2º- _wandbruntime‚7876  _runtime‚7876% '»…N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156447690 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156437531 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86057924 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10154629729792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10097889481728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10154741283328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1323176176981152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86057924 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15179583 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35956390 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35956097 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1341059146110976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10097548050432 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156437531 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚8699 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1330961239589888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156447690 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7876.196693936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70389764 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20776514 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1333258825301089 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚289303930461184¢èzZ'¼ŠN _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1330961256629248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156447690 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7876.196693936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70389764 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20776514 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1333258825301089 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚289303930461184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156447690 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156437531 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86057924 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10154629729792 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10097889481728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10154741283328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1323176176981152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86057924 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15179583 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35956390 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35956097 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1341059146110976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10097548050432 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156437531 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚8699 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1330961239589888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1348378697248256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20888400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1340537302832261 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301139204881920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10209441186816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚8749 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10209333820416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290929745082880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10152348477440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20888098 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _step‚718 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157294384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1330400262647920 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86532906 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1338226707234304 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1340536945231627 % _timestamp‚1.7845147460821655e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86532906 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301139078903296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10136682583707 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290929763695104 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1338226690194944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70771621 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70771621 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10152007053312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86522763 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10137023144981 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1348378697248256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20888400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1340537302832261 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301139204881920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10209441186816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚8749 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10209333820416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290929745082880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10152348477440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20888098 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _step‚718 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157294384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1330400262647920 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86532906 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1338226707234304 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1340536945231627 % _timestamp‚1.7845147460821655e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86532906 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301139078903296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10136682583707 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290929763695104 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1338226690194944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70771621 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70771621 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10152007053312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86522763 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10137023144981 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36156670 _runtime‚7920.283073904 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1340685841888097 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70779414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10210449325056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1330547710354752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1348528440320000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10153458583552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚301173500649472 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10138131533345 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290962925352960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86532419 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚8749 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70779414 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157311833 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157321992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20890387 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15266286 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70779416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1338374981736448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7845147467073104e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157311833 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1340685484287351 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1338374964697088 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1348528081849344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10210556684288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157321992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15266283 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1330547693315392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20890663 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10137790971959 _step‚719 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86532419 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157321992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20890387 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15266286 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70779416 C9DeviceStatsMonitor.on_tra d ?çin_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1338374981736448 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7845147467073104e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157311833 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1340685484287351 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1338374964697088 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1348528081849344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10210556684288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157321992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15266283 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1330547693315392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20890663 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10137790971959 _step‚719 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86532419 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36156670 _runtime‚7920.283073904 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1340685841888097 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70779414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10210449325056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1330547710354752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1348528440320000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10153458583552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚301173500649472 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10138131533345 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290962925352960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180Ï®ðRú2Ï- _wandbruntime‚7920  _runtime‚7920x’¸q Ð… %loss/total_loss‚7.932977614345663 epoch‚21 _step‚720 3loss/SRqqt2/assignment_loss‚0.48638618286032986 2loss/FBt1/assignment_loss‚0.015446711722059013 /loss/FRt1/detection_loss‚0.4446736989228199 /loss/FRt2/detection_loss‚1.3877311725083292 0loss/FRt1/assignment_loss‚1.7226043458791658 2loss/SRqqt1/assignment_loss‚0.4902398366397737 1loss/SRqqt2/detection_loss‚0.6431535024618813 trainer/global_step‚8749 % _timestamp‚1.7845147467083523e+09 2loss/FBt2/assignment_loss‚0.011284192822758199 _runtime‚7920.283622407 /loss/FRt2/assignment_loss‚0.804856983990204 1loss/SRqqt1/detection_loss‚0.8891383869536085 /loss/FBt1/detection_loss‚0.6141997105836506 0loss/FBt2/detection_loss‚0.42326288900108305ÏÓ©ºÑŠ %loss/total_loss‚7.932977614345663 epoch‚21 _step‚720 3loss/SRqqt2/assignment_loss‚0.48638618286032986 2loss/FBt1/assignment_loss‚0.015446711722059013 /loss/FRt1/detection_loss‚0.4446736989228199 /loss/FRt2/detection_loss‚1.3877311725083292 0loss/FRt1/assignment_loss‚1.7226043458791658 2loss/SRqqt1/assignment_loss‚0.4902398366397737 1loss/SRqqt2/detection_loss‚0.6431535024618813 trainer/global_step‚8749 % _timestamp‚1.7845147467083523e+09 2loss/FBt2/assignment_loss‚0.011284192822758199 _runtime‚7920.283622407 /loss/FRt2/assignment_loss‚0.804856983990204 1loss/SRqqt1/detection_loss‚0.8891383869536085 /loss/FBt1/detection_loss‚0.6141997105836506 0loss/FBt2/detection_loss‚0.42326288900108305ÐãMätUÒ‚(:Ê É‘öÒå¿ÇË cpu‚25.69352006903999# disk.dm-1.out‚11768.13134765625 disk.nvme0n1.in‚ 0.00390625& disk.nvme1c1n1.out‚24779.705078125 proc.memory.availableMB‚530* proc.memory.percent‚18.428921699523926% disk./.usageGB‚1166.6257209777832 disk.md0.out‚237.0625" disk.dm-0.out‚59386.7724609375( disk.nvme0c0n1.out‚46882.26025390625& disk.nvme0n1.out‚46882.26025390625$ disk.nvme1n1.out‚24779.705078125! memory_percent‚96.76513671875 proc.cpu.threads‚17 network.sent‚ 1.3954598e+07( disk./.usagePercent‚75.9892995263171 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0 network.recv‚ 2.122624e+06 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0# proc.memory.rssMB‚ 3019.39453125OœØŸÓ‚(:” É‘öÒ«ÏüË gpu.0.gpu‚29.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.104' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.034666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aÃTÔ‚(:É Ø‘öÒûÿÙË network.recv‚ 2.127005e+06( disk./.usagePercent‚75.9893226344002 disk.md0.in‚0 disk.md0.out‚ 237.4765625# proc.memory.rssMB‚ 3019.39453125" disk.dm-0.out‚59387.5888671875( disk.nvme0c0n1.out‚46883.42626953125 disk.nvme0n1.in‚ 0.00390625! memory_percent‚96.76513671875 proc.cpu.threads‚17 network.sent‚ 1.3995969e+07$ disk.nvme1c1n1.out‚ 24780.2890625" disk.nvme1n1.out‚ 24780.2890625 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0& disk.nvme0n1.out‚46883.42626953125* proc.memory.percent‚18.428921699523926$ disk./.usageGB‚1166.626075744629 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 proc.memory.availableMB‚530 cpu‚25.693154705925004# disk.dm-1.out‚11768.178222656253#“Õ‚(:ˆ Ø‘öÒó¥¤Ì gpu.0.gpu‚57.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.758' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.586 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³g 2Ö- _wandbruntime‚7963  _runtime‚7963kËÅCÉ×à % _timestamp‚1.7845147902100246e+09 _runtime‚7963.785599101 _step‚721 ' lr-AdamW/pg1‚0.00037566110219328314 ' lr-AdamW/pg2‚0.00037566110219328314 trainer/global_step‚8799¹g»§ÎØÈ ' lr-AdamW/pg1‚0.00037566110219328314 ' lr-AdamW/pg2‚0.00037566110219328314 trainer/global_step‚8799 % _timestamp‚1.7845147902100246e+09 _runtime‚7963.785599101 _step‚721Ñôê·2Ù- _wandbruntime‚7963  _runtime‚7963bZ(ÚûO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71161271 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1345640415302144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21002501 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚8799 _runtime‚7963.786778085 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15351249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36354028 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1345640432341504 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1355847991457280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158178831 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158168686 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1355848349920768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71161271 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1347963961818635 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10192506358389 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87017558 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚292588777199104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302853895281152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _step‚722 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36353750 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1337771813060880 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1355848349920768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87017558 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1337771796021520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71161273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10207917579264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15351252 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1355847991457280 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10207917579264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21002776 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1345640415302144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10207576155136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10207576155136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1355847991457280 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10207917579264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21002776 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1345640415302144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10207576155136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10207576155136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15351249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36354028 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1345640432341504 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1355847991457280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158178831 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158168686 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1355848349920768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71161271 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1347963961818635 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10192506358389 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87017558 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚292588777199104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302853895281152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _step‚722 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36353750 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1337771813060880 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1355848349920768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87017558 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1337771796021520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71161273 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10207917579264 JBDeviceStatsMonitor.on_train_batch_start/inactive_splÏÆsit.all.current‚278 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15351252ÒXµa2Ü- _wandbruntime‚7964  _runtime‚7964ša° 'Ý„N B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71169066 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158196294 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1345788706843648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1345788689804288 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1345788706843648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87017071 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10208686254080 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10193274185367 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1345788689804288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚7964.411996453 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1337919226688992 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10266354053120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1355997376058368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87027228 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15352986 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1355997734529024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15352983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158196294 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71169064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87017071 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10193614746753 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158186135 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10209027685376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158186135 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1348112858475105 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71169066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36357730 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21004747 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36358021 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1337919243728352,0•'Þ‰N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10193274185367 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1345788689804288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚7964.411996453 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1337919226688992 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10266354053120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1355997376058368 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87027228 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15352986 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1355997734529024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15352983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158196294 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71169064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87017071 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10193614746753 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158186135 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10209027685376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158186135 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1348112858475105 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71169066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36357730 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21004747 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36358021 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1337919243728352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71169066 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158196294 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1345788706843648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1345788689804288 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1345788706843648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87017071 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10208686254080Ó,Âø]2ß- _wandbruntime‚7964  _runtime‚7964É?/Ãà‚ /loss/FBt1/detection_loss‚0.4533649532060171 % _timestamp‚1.7845147908373554e+09 _step‚724 0loss/FRt2/assignment_loss‚0.6872887571961963 1loss/FBt1/assignment_loss‚0.03240504522168859 /loss/FBt2/detection_loss‚0.5045328004258719 0loss/FRt1/assignment_loss‚1.7046513870406177 3loss/SRqqt1/assignment_loss‚0.47909399009948783 2loss/SRqqt2/assignment_loss‚0.5966920178500611 2loss/FBt2/assignment_loss‚0.014223879749141086 1loss/SRqqt2/detection_loss‚0.6400888802632984 %loss/total_loss‚7.781899536235076 epoch‚21 trainer/global_step‚8799 /loss/FRt1/detection_loss‚0.4897390753627772 -loss/FRt2/detection_loss‚1.33480206607879 1loss/SRqqt1/detection_loss‚0.8450166837411293 _runtime‚7964.412649075'Ô ) ᇠ1loss/SRqqt1/detection_loss‚0.8450166837411293 _runtime‚7964.412649075 /loss/FBt1/detection_loss‚0.4533649532060171 % _timestamp‚1.7845147908373554e+09 _step‚724 0loss/FRt2/assignment_loss‚0.6872887571961963 1loss/FBt1/assignment_loss‚0.03240504522168859 /loss/FBt2/detection_loss‚0.5045328004258719 0loss/FRt1/assignment_loss‚1.7046513870406177 3loss/SRqqt1/assignment_loss‚0.47909399009948783 2loss/SRqqt2/assignment_loss‚0.5966920178500611 2loss/FBt2/assignment_loss‚0.014223879749141086 1loss/SRqqt2/detection_loss‚0.6400888802632984 %loss/total_loss‚7.781899536235076 epoch‚21 trainer/global_step‚8799 /loss/FRt1/detection_loss‚0.4897390753627772 -loss/FRt2/detection_loss‚1.33480206607879Ô»"gâ‚(:„ ç‘öÒȳ±Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.09' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.03 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wñÈóhã‚(:Ý ç‘öÒúŠðÍ disk.rbd0.in‚0# proc.memory.rssMB‚ 3019.39453125* proc.memory.percent‚18.428921699523926 network.sent‚ 1.3998891e+07 network.recv‚ 2.130367e+06 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚24781.01904296875 disk.nvme1n1.in‚0 disk.nvme0c0n1.in‚ 0.00390625 cpu‚25.692683261866502 proc.cpu.threads‚17% disk./.usageGB‚1166.6261520385742 disk.nvme0n1.in‚ 0.00390625 disk.md0.out‚ 237.8515625 disk.dm-1.in‚0' disk.nvme0c0n1.out‚46885.5810546875$ memory_percent‚96.76826000213623( proc.memory.availableMB‚ 529.48828125 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0) disk./.usagePercent‚75.98932760388043% disk.nvme0n1.out‚46885.5810546875& disk.nvme1n1.out‚24781.01904296875# disk.dm-0.out‚59389.55712890625" disk.dm-1.out‚11768.2919921875 UŒ^ä‚(:Ó ö‘öÒÅƽË# proc.memory.rssMB‚ 3020.71484375) proc.memory.percent‚18.43698024749756 cpu‚25.69727758386459 network.sent‚ 1.4063127e+07) disk./.usagePercent‚75.98932014966007% disk.nvme0c0n1.out‚46886.84765625# disk.nvme0n1.out‚46886.84765625 disk.dm-0.in‚ 0.00390625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0% proc.memory.availableMB‚ 527.65625 proc.cpu.threads‚17 network.recv‚ 2.134205e+06% disk./.usageGB‚1166.6260375976562 disk.rbd0.in‚0( disk.nvme1c1n1.out‚24781.57177734375 disk.nvme1n1.in‚0& disk.nvme1n1.out‚24781.57177734375 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.md0.out‚ 238.1796875# disk.dm-0.out‚59390.57958984375" disk.dm-1.out‚11768.3740234375# memory_percent‚96.7794418334961…ÀÎNžå‚(:“ ö‘öÒ‘ÝþË gpu.0.gpu‚97.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚287.809' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.93633333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ì:2æ- _wandbruntime‚7980  _runtime‚7980ÅE”¹{ ç¹z 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.20291416347026825 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5789345502853394 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4775286316871643 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5447150468826294 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.768449604511261 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9911012053489685 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 *jet/accuracy_2_of_4‚0.6657357215881348 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.594664990901947 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994933605194092 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22838127613067627 5 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.86443692445755 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 /particle/accuracy_3_of_3‚0.9994120597839355 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8904174566268921 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2382926493883133 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.64336097240448 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.708014965057373 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4753407835960388 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3671875 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24136734008789062 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21688275039196014 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_3_of_5‚1 _step‚725 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6146506667137146 _runtime‚7980.631971411 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8697279095649719 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02915692888200283 jet/accuracy_5_of_5‚0 trainer/global_step‚8799 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 +jet/accuracy_3_of_4‚0.09808392822742462 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2275940179824829 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5976858735084534 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17315202951431274 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6103208065032959 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.056297119706869125 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2927859127521515 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18329018354415894 +jet/accuracy_1_of_1‚0.29692786931991577 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968207478523254 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.982202410697937 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5575000047683716 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.690105140209198 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5446922779083252 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.13328063488006592 *jet/accuracy_1_of_4‚0.9739487171173096 /particle/accuracy_4_of_4‚0.9451785087585449 +particle/specificity‚0.9118961095809937 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9251580834388733 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22336600720882416 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.23000285029411316 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5418403744697571 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5476891398429871 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5110988020896912 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.258195161819458 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6059011220932007 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33016708493232727 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5829927921295166 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9965075254440308 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_2‚0.7658746838569641 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4218750298023224 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976294636726379 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6598281860351562 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8606576323509216 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22635871171951294 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8741875886917114 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5432425737380981 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5432425737380981 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9996392130851746 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3233288824558258 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5784589648246765 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25085338950157166 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2910592555999756 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10273154824972153 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.553964376449585 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6487738490104675 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5647522211074829 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03695188835263252 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.624265193939209 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_2_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25548359751701355 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.573039710521698 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997321367263794 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26616644859313965 jet/accuracy_1_of_5‚1 &¿¢±+Ä Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_3_of_4‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23948781192302704 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884513020515442 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.018900182098150253 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6064738035202026 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10083889961242676 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7202003002166748 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8888888955116272 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6953125 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930664896965027 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7071952819824219 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21933214366436005 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5508977770805359 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5911351442337036 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2960323393344879 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.23000285029411316 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8701813817024231 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23597785830497742 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941222667694092 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31928277015686035 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5760399699211121 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8751012682914734 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23615901172161102 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1855470985174179 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24121466279029846 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6445221304893494 particle/accuracy_5_of_5‚0 % _timestamp‚1.7845148070558567e+09 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2820146083831787 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6283138990402222 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6286982893943787 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9527338743209839 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04910745099186897 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2275940179824829 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.229007288813591 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6285510659217834 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7436144948005676 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_2_of_3‚0.45320823788642883 (particle/accuracy‚0.8510704040527344 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6170295476913452 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1854390949010849 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5624397397041321 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.22565793991088867 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22608636319637299 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5324376225471497 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8888888955116272 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9986771941184998 ,"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6133726239204407 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9921170473098755 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 *jet/accuracy_1_of_3‚0.9051002264022827 +jet/accuracy_3_of_3‚0.07878696918487549 epoch‚21 *loss/val_total_loss‚7.5930778320332335 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24287641048431396 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880818128585815 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9190275073051453 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47963881492614746 particle/accuracy_2_of_4‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17500001192092896 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5419429540634155 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7851474285125732 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3418525457382202 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03643161803483963 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_2_of_5‚1 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2013307362794876 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7129287123680115 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5911351442337036 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9969481229782104 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2324335277080536 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5369387865066528,ÁD=è¾z 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9527338743209839 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04910745099186897 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2275940179824829 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.229007288813591 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6285510659217834 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7436144948005676 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_2_of_3‚0.45320823788642883 (particle/accuracy‚0.8510704040527344 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6170295476913452 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1854390949010849 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5624397397041321 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.22565793991088867 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22608636319637299 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5324376225471497 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8888888955116272 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9986771941184998 ,"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6133726239204407 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9921170473098755 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 *jet/accuracy_1_of_3‚0.9051002264022827 +jet/accuracy_3_of_3‚0.07878696918487549 epoch‚21 *loss/val_total_loss‚7.5930778320332335 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24287641048431396 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880818128585815 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9190275073051453 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47963881492614746 particle/accuracy_2_of_4‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17500001192092896 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5419429540634155 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7851474285125732 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3418525457382202 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03643161803483963 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_2_of_5‚1 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2013307362794876 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7129287123680115 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5911351442337036 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9969481229782104 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2324335277080536 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5369387865066528 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.20291416347026825 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5789345502853394 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4775286316871643 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5447150468826294 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.768449604511261 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9911012053489685 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 *jet/accuracy_2_of_4‚0.6657357215881348 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.594664990901947 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994933605194092 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22838127613067627 5 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.86443692445755 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 /particle/accuracy_3_of_3‚0.9994120597839355 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8904174566268921 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2382926493883133 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.64336097240448 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_2‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.708014965057373 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4753407835960388 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3671875 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24136734008789062 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21688275039196014 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_3_of_5‚1 _step‚725 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6146506667137146 _runtime‚7980.631971411 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8697279095649719 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02915692888200283 jet/accuracy_5_of_5‚0 trainer/global_step‚8799 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 +jet/accuracy_3_of_4‚0.09808392822742462 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2275940179824829 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5976858735084534 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17315202951431274 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6103208065032959 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.056297119706869125 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2927859127521515 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18329018354415894 +jet/accuracy_1_of_1‚0.29692786931991577 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968207478523254 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.982202410697937 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5575000047683716 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.690105140209198 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5446922779083252 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.13328063488006592 *jet/accuracy_1_of_4‚0.9739487171173096 /particle/accuracy_4_of_4‚0.9451785087585449 +particle/specificity‚0.9118961095809937 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9251580834388733 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.22336600720882416 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.23000285029411316 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5418403744697571 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5476891398429871 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5110988020896912 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.258195161819458 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6059011220932007 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33016708493232727 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5829927921295166 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9965075254440308 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_2‚0.7658746838569641 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4218750298023224 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976294636726379 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6598281860351562 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8606576323509216 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22635871171951294 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8741875886917114 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5432425737380981 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5432425737380981 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9996392130851746 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3233288824558258 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5784589648246765 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25085338950157166 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2910592555999756 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10273154824972153 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.553964376449585 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6487738490104675 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5647522211074829 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03695188835263252 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.624265193939209 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_2_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25548359751701355 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.573039710521698 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997321367263794 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26616644859313965 jet/accuracy_1_of_5‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_3_of_4‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23948781192302704 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884513020515442 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.018900182098150253 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6064738035202026 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10083889961242676 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7202003002166748 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8888888955116272 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6953125 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930664896965027 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.7071952819824219 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21933214366436005 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5508977770805359 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5911351442337036 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2960323393344879 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.23000285029411316 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8701813817024231 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23597785830497742 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941222667694092 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31928277015686035 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5760399699211121 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8751012682914734 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23615901172161102 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1855470985174179 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24121466279029846 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6445221304893494 particle/accuracy_5_of_5‚0 % _timestamp‚1.7845148070558567e+09 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2820146083831787 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6283138990402222 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6286982893943787Õpà*dé‚(:Ù …’öÒ±óÁË disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625) proc.memory.percent‚18.43698024749756 disk.dm-1.in‚0# disk.dm-1.out‚11768.54248046875 disk.rbd0.out‚0 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 238.5703125( proc.memory.availableMB‚ 527.34765625# proc.memory.rssMB‚ 3020.71484375 network.sent‚ 1.4066049e+07 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚46889.6923828125' disk.nvme1c1n1.out‚24782.4384765625# disk.dm-0.out‚59393.28076171875" memory_percent‚96.781325340271 proc.cpu.threads‚17 network.recv‚ 2.137792e+06) disk./.usagePercent‚75.98936189329405 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚46889.6923828125% disk.nvme1n1.out‚24782.4384765625 cpu‚25.697064014831604% disk./.usageGB‚1166.6266784667969%žê‚(:“ …’öÒ¿žëË gpu.0.gpu‚44.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.96' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.986666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 6KSë‚(:È ”’öÒØä´Ë( proc.memory.availableMB‚ 527.30859375 cpu‚25.69660295470909 network.sent‚ 1.4124766e+07( disk./.usagePercent‚75.9893671112483 disk.nvme1n1.in‚0 disk.nvme1n1.out‚ 24783.3125 disk.dm-1.in‚0# memory_percent‚96.7815637588501) proc.memory.percent‚18.43698024749756 proc.cpu.threads‚17 network.recv‚ 2.142164e+06 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚59394.55615234375# disk.dm-1.out‚11768.60107421875$ disk.nvme0n1.out‚46891.384765625% disk./.usageGB‚1166.6267585754395 disk.md0.in‚0 disk.md0.out‚ 239.15625 disk.rbd0.in‚0# proc.memory.rssMB‚ 3020.71484375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0c0n1.out‚46891.384765625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0! disk.nvme1c1n1.out‚ 24783.3125ôΞì‚(:“ ”’öÒØÉ÷Ë gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚279.161' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.05366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“¸äl2í- _wandbruntime‚8023  _runtime‚8023çäáàÅî¿ trainer/global_step‚8849 % _timestamp‚1.7845148503102798e+09 _runtime‚8023.885864347 _step‚726 % lr-AdamW/pg1‚0.000373258413510071 % lr-AdamW/pg2‚0.000373258413510071‘#ÊïÄ _step‚726 % lr-AdamW/pg1‚0.000373258413510071 % lr-AdamW/pg2‚0.000373258413510071 trainer/global_step‚8849 % _timestamp‚1.7845148503102798e+09 _runtime‚8023.885864347ÖlŠ2ð- _wandbruntime‚8023  _runtime‚8023»Ë¾ÐÅñûO VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1365279947377664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10268293420694 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87583704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36598050 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71673729 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1365280305841152 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1365280305841152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71673729 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15442068 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10341884230656 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10283804516864 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10283804516864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10341783155712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87573561 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87583704 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10283463092736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚294270369365504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159247290 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1365279947377664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 % _timestamp‚1.7845148503112543e+09 J‡¨(ò€P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159257435 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10283463092736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚294270369365504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159247290 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1365279947377664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 % _timestamp‚1.7845148503112543e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1365279947377664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10268293420694 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87583704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36598050 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71673729 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1365280305841152 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1365280305841152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71673729 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15442068 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10341884230656 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10283804516864 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10283804516864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10341783155712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87573561 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87583704 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36601709 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1355144775826432 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1355144758787072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159274898 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1355144775826432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10284914622976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36601998 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1347212016070816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _step‚728 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10284573191680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304646546856960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71681522 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10342997221376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71681522 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87583217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10269061247672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10342896153600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15443799 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1365429331978752 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8849 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10284573191680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159264739 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159264739 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1357481077318488 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496¤×_ä^"õŠN UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15443799 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1365429331978752 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8849 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10284573191680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159264739 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159264739 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1357481077318488 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36601709 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1355144775826432 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1355144758787072 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159274898 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1355144775826432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10284914622976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36601998 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1347212016070816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _step‚728 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10284573191680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304646546856960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 J=DevicŠ¤Ÿ¦²eStatsMonitor.on_train_batch_end/active.large_pool.freed‚71681522 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10342997221376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71681522 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87583217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10269061247672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10342896153600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647Ø´M`2ö- _wandbruntime‚8024  _runtime‚8024…4…¢÷ 2loss/FBt1/assignment_loss‚0.002189397426229526 .loss/FRt1/detection_loss‚0.434958276107032 1loss/SRqqt1/detection_loss‚0.8661999345190438 1loss/SRqqt2/detection_loss‚0.5853571968965984 % _timestamp‚1.7845148509353583e+09 _runtime‚ 8024.5106761 _step‚729 /loss/FRt1/assignment_loss‚1.775733963529919 2loss/SRqqt2/assignment_loss‚0.5491533187477002 2loss/SRqqt1/assignment_loss‚0.5466518599439592 4loss/FBt2/assignment_loss‚0.00012820302781774476 /loss/FRt2/detection_loss‚1.3658407510682684 /loss/FBt1/detection_loss‚0.3568132584054104 /loss/FRt2/assignment_loss‚0.760393525946475 /loss/FBt2/detection_loss‚0.3574040784878889 %loss/total_loss‚7.600823764106343 epoch‚22 trainer/global_step‚88499v ø† /loss/FBt2/detection_loss‚0.3574040784878889 %loss/total_loss‚7.600823764106343 epoch‚22 trainer/global_step‚8849 2loss/FBt1/assignment_loss‚0.002189397426229526 .loss/FRt1/detection_loss‚0.434958276107032 1loss/SRqqt1/detection_loss‚0.8661999345190438 1loss/SRqqt2/detection_loss‚0.5853571968965984 % _timestamp‚1.7845148509353583e+09 _runtime‚ 8024.5106761 _step‚729 /loss/FRt1/assignment_loss‚1.775733963529919 2loss/SRqqt2/assignment_loss‚0.5491533187477002 2loss/SRqqt1/assignment_loss‚0.5466518599439592 4loss/FBt2/assignment_loss‚0.00012820302781774476 /loss/FRt2/detection_loss‚1.3658407510682684 /loss/FBt1/detection_loss‚0.3568132584054104 /loss/FRt2/assignment_loss‚0.760393525946475Ù}ÇJèjù‚(:ß £’öÒ¿ÈÐË disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚75.98934971806747( proc.memory.availableMB‚ 526.81640625# proc.memory.rssMB‚ 3020.71484375 cpu‚25.696110403243132 network.sent‚ 1.4140531e+07 network.recv‚ 2.145569e+06% disk./.usageGB‚1166.6264915466309( disk.nvme0c0n1.out‚46894.07177734375 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0# disk.dm-0.out‚59397.04638671875# disk.dm-1.out‚11768.67529296875$ memory_percent‚96.78456783294678) proc.memory.percent‚18.43698024749756 proc.cpu.threads‚17& disk.nvme0n1.out‚46894.07177734375( disk.nvme1c1n1.out‚24784.46630859375& disk.nvme1n1.out‚24784.46630859375 disk.md0.out‚ 239.7578125 disk.dm-0.in‚ 0.00390625KÇ÷Ýœú‚(:‘ £’öÒðØÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.92' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.973333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„6§ãbû‚(:× ²’öÒà‘¿Ë disk.nvme1n1.in‚0) proc.memory.percent‚18.43698024749756 cpu‚25.69568411686579& disk.nvme0c0n1.out‚46895.357421875 network.sent‚ 1.4191927e+07) disk./.usagePercent‚75.98934723332735 disk.md0.in‚0 disk.md0.out‚ 240.2734375! disk.dm-0.out‚59397.904296875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625 network.recv‚ 2.149525e+06 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚24788.41357421875& disk.nvme1n1.out‚24788.41357421875# memory_percent‚96.7860460281372( proc.memory.availableMB‚ 526.57421875# proc.memory.rssMB‚ 3020.71484375 disk.dm-0.in‚ 0.00390625 disk.nvme1c1n1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1166.6264533996582# disk.dm-1.out‚11771.96044921875 disk.rbd0.in‚0$ disk.nvme0n1.out‚46895.357421875—!÷‹“ü‚(:ˆ ²’öÒ‰õøË gpu.0.gpu‚41.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.939' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.313 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±x.aý‚(:Ö Á’öÒ…¥ÎË& disk.nvme1n1.out‚24789.57470703125 network.recv‚ 2.153109e+06) disk./.usagePercent‚75.98934673637933 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 network.sent‚ 1.4194891e+07% disk./.usageGB‚1166.6264457702637 disk.rbd0.in‚0( proc.memory.availableMB‚ 526.33203125# proc.memory.rssMB‚ 3020.71484375) proc.memory.percent‚18.43698024749756 proc.cpu.threads‚17 disk.md0.out‚240.9375$ memory_percent‚96.78752422332764 cpu‚25.695195745223877 disk.md0.in‚0# disk.dm-0.out‚59400.72607421875! disk.dm-1.out‚11772.107421875& disk.nvme0c0n1.out‚46898.583984375$ disk.nvme0n1.out‚46898.583984375 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚24789.57470703125– Åžþ‚(:“ Á’öÒŒàåË gpu.0.gpu‚89.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.776' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.59200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vÒ2ÿ- _wandbruntime‚8067  _runtime‚8067ëáCŸÆ€À & lr-AdamW/pg1‚0.0003708514848900993 & lr-AdamW/pg2‚0.0003708514848900993 trainer/global_step‚8899 $ _timestamp‚1.784514894264323e+09 _runtime‚8067.839945365 _step‚730tÂàËÅ _step‚730 & lr-AdamW/pg1‚0.0003708514848900993 & lr-AdamW/pg2‚0.0003708514848900993 trainer/global_step‚8899 $ _timestamp‚1.784514894264323e+09 _runtime‚8067.839945365Úa ÌK2‚- _wandbruntime‚8067  _runtime‚8067@ ý‘ÿ'ƒùO A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88058213 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15528765 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚306327030016000 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1364759912450406 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72063379 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1362410226431488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1372749600050176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1372749241586688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 _step‚731 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36798969 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72063381 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10397665758720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10339032194560 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1362410209392128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1354436135816304 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1372749600050176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21270500 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1362410226431488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10339032194560 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10397773125120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160121592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295929364257280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1372749241586688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 _step‚731 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36798969 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72063381 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10397665758720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10339032194560 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1362410209392128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1354436135816304 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1372749600050176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21270500 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1362410226431488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10339032194560 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10397773125120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160121592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295929364257280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1372749600050176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1362558500933632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72071174 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15530499 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10398786250240 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1362558483894272 _runtime‚8068.463116307 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21272489 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚306361450457600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚732 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚306361330777600 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1362558483894272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36803277 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160139041 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845148948873231e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36802988 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1362558500933632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10340483724800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch[ó…Ϥ_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72071174 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1372898984658432 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1372898626187776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1354583566483776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72071172 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1364908093905496 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88078026 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10340142293504 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72071172 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160149200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160149200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15530502 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10324544461080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1372898984658432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88067869 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21272775 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10340483724800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295962544527360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚8899 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36802988 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1362558500933632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10340483724800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72071174 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1372898984658432 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1372898626187776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1354583566483776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72071172 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1364908093905496 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88078026 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10340142293504 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72071172 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160149200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160149200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15530502 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10324544461080 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1372898984658432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88067869 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21272775 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10340483724800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295962544527360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚8899 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1362558500933632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72071174 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15530499 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10398786250240 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1362558483894272 _runtime‚8068.463116307 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21272489 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚306361450457600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚732 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚306361330777600 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1362558483894272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36803277 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160139041 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845148948873231e+09ܨRÊÃ2ˆ- _wandbruntime‚8068  _runtime‚8068'6ˆî ‰† 0loss/FBt2/detection_loss‚0.45253760678492283 _runtime‚8068.463659158 _step‚733 /loss/FRt1/assignment_loss‚1.824350138357345 3loss/FBt2/assignment_loss‚0.0003713851202528846 0loss/SRqqt1/detection_loss‚0.774011622170969 epoch‚22 trainer/global_step‚8899 $ _timestamp‚1.784514894888369e+09 0loss/FRt2/assignment_loss‚0.8054503794908524 2loss/SRqqt2/assignment_loss‚0.5209580012095221 1loss/SRqqt2/detection_loss‚0.5722228942226321 0loss/FBt1/detection_loss‚0.45924964764465886 %loss/total_loss‚7.892849973987223 0loss/FRt1/detection_loss‚0.48368331331564557 /loss/FRt2/detection_loss‚1.4932650942658074 2loss/SRqqt1/assignment_loss‚0.5063555203874331 3loss/FBt1/assignment_loss‚0.0003943710171822349 Z•°Š‹ 3loss/FBt2/assignment_loss‚0.0003713851202528846 0loss/SRqqt1/detection_loss‚0.774011622170969 epoch‚22 trainer/global_step‚8899 $ _timestamp‚1.784514894888369e+09 0loss/FRt2/assignment_loss‚0.8054503794908524 2loss/SRqqt2/assignment_loss‚0.5209580012095221 1loss/SRqqt2/detection_loss‚0.5722228942226321 0loss/FBt1/detection_loss‚0.45924964764465886 %loss/total_loss‚7.892849973987223 0loss/FRt1/detection_loss‚0.48368331331564557 /loss/FRt2/detection_loss‚1.4932650942658074 2loss/SRqqt1/assignment_loss‚0.5063555203874331 3loss/FBt1/assignment_loss‚0.0003943710171822349 0loss/FBt2/detection_loss‚0.45253760678492283 _runtime‚8068.463659158 _step‚733 /loss/FRt1/assignment_loss‚1.824350138357345ÝÜ`‘N^‹‚(:Ó Ð’öÒ÷£èË disk.rbd0.in‚0) proc.memory.percent‚18.43698024749756 network.sent‚ 1.421253e+07 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0( disk.nvme1c1n1.out‚24790.25634765625& proc.memory.availableMB‚ 526.328125 network.recv‚ 2.156355e+06% disk./.usageGB‚1166.6267318725586 disk.md0.out‚ 241.44921875 disk.dm-1.in‚0! disk.dm-1.out‚11772.169921875 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 46899.8515625# proc.memory.rssMB‚ 3020.71484375) disk./.usagePercent‚75.98936537193022& disk.nvme1n1.out‚24790.25634765625# disk.dm-0.out‚59401.53076171875 disk.nvme0c0n1.in‚ 0.00390625" disk.nvme0n1.out‚ 46899.8515625$ memory_percent‚96.78754806518555 cpu‚25.694770475932245 proc.cpu.threads‚17S˜3Ç“Œ‚(:ˆ Ð’öÒÎú­Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚142.998' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚47.666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ÁS]‚(:Ò ß’öÒ’ûâË disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.md0.out‚241.9375$ memory_percent‚96.78709506988525( proc.memory.availableMB‚ 526.40234375# proc.memory.rssMB‚ 3020.71484375 proc.cpu.threads‚17 network.recv‚ 2.160917e+06$ disk./.usageGB‚1166.627239227295 network.sent‚ 1.4263928e+07% disk.nvme0n1.out‚46902.4794921875& disk.nvme1c1n1.out‚24791.154296875$ disk.nvme1n1.out‚24791.154296875" disk.dm-1.out‚11772.4033203125 disk.rbd0.out‚0) proc.memory.percent‚18.43698024749756 cpu‚25.694379271229714) disk./.usagePercent‚75.98939841897378 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625! disk.dm-0.out‚59403.775390625 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚46902.4794921875æ·M4žŽ‚(:“ ß’öÒ¤³­Ì gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚272.122' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.70733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕaÓé_‚(:Ô î’öÒµê¶Ë disk.md0.out‚ 242.30078125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚46903.3388671875 disk.nvme0n1.in‚ 0.00390625# proc.memory.rssMB‚ 3020.71484375 cpu‚25.694022302636835 network.sent‚ 1.4268773e+07 disk.nvme1n1.in‚0 disk.dm-0.in‚ 0.00390625! disk.dm-0.out‚59404.357421875 disk.dm-1.in‚0% disk.nvme0n1.out‚46903.3388671875$ memory_percent‚96.78592681884766% proc.memory.availableMB‚ 526.59375) disk./.usagePercent‚75.98934102147706% disk./.usageGB‚1166.6263580322266 disk.nvme1c1n1.in‚0$ disk.nvme1n1.out‚24791.732421875" disk.dm-1.out‚11772.4775390625 disk.rbd0.out‚0) proc.memory.percent‚18.43698024749756 proc.cpu.threads‚17 network.recv‚ 2.164597e+06& disk.nvme1c1n1.out‚24791.732421875 disk.md0.in‚0 disk.nvme0c0n1.in‚ 0.00390625áM|“‚(:ˆ î’öÒØÓúË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚121.347' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.449 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à‡æÄ2‘- _wandbruntime‚8111  _runtime‚8111æ ¢ìÉ’Ã % _timestamp‚1.7845149377817574e+09 _runtime‚8111.357353461 _step‚734 ' lr-AdamW/pg1‚0.00036844047092589576 ' lr-AdamW/pg2‚0.00036844047092589576 trainer/global_step‚8949åàðÅÎ“È _runtime‚8111.357353461 _step‚734 ' lr-AdamW/pg1‚0.00036844047092589576 ' lr-AdamW/pg2‚0.00036844047092589576 trainer/global_step‚8949 % _timestamp‚1.7845149377817574e+09Þ|M2”- _wandbruntime‚8111  _runtime‚8111´åi#(•ûO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10394601296384 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1380218535795712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10453513022464 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1369823951538688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21384689 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15615468 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚8949 % _timestamp‚1.7845149377828717e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72453031 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1361807652150544 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1369823951538688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37000154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88553008 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _runtime‚8111.362056522 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1369823934499328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88553008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37000446 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10453620388864 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1372186929037414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72453029 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21384978 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _step‚735 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160995894 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88542865 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1380218894259200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72453031 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1380218894259200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚297588377761280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1369823934499328 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10394942720512 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚297588396373504 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10379259847510 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10378919286236 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15615465 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1380218535795712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161006039 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72453029 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚308041890783744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1361807669189904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1361807669189904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1380218535795712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10453513022464 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1369823951538688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21384689 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15615468 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚8949 % _timestamp‚1.7845149377828717e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72453031 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1361807652150544 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1369823951538688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37000154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88553008 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _runtime‚8111.362056522 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1369823934499328 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88553008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37000446 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10453620388864 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1372186929037414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72453029 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21384978 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _step‚735 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160995894 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88542865 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1380218894259200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72453031 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1380218894259200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚297588377761280 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145ßiÔQ2—- _wandbruntime‚8111  _runtime‚8111…&Æ '˜…N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161013343 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308076314518528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1369972226040832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37004426 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15617202 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15617199 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161023502 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21386949 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845149384058623e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88562678 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10396052826624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10395711395328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1372335468093250 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161023502 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚297621558031360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1361955082818016 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161013343 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1361955099857376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1380367920396800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚8949 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72460824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1369972209001472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88562678 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚8111.981630009 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1372335110492504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10380368235874 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88552521 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1369972226040832 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1369972209001472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21387224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37004148 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 _step‚736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10396052826624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10454737874944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1380368278867456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308076184352768 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1380368278867456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10395711395328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303ЇoK'™ŠN D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161013343 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1361955099857376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1380367920396800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚8949 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72460824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1369972209001472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88562678 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚8111.981630009 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1372335110492504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10380368235874 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88552521 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1369972226040832 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1369972209001472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21387224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37004148 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 _step‚736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10396052826624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10454737874944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1380368278867456 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308076184352768 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1380368278867456 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10395711395328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161013343 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚308076314518528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1369972226040832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37004426 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15617202 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15617199 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161023502 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21386949 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845149384058623e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88562678 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10396052826624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10395711395328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1372335468093250 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161023502 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚297621558031360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1361955082818016 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452àŠ˜/2š- _wandbruntime‚8111  _runtime‚8111eùo° ›„ 0loss/FRt2/assignment_loss‚0.7146945302603674 /loss/FBt1/detection_loss‚0.7971126196874927 /loss/FBt2/detection_loss‚0.3574364054665873 trainer/global_step‚8949 2loss/SRqqt1/assignment_loss‚0.4399427823415489 2loss/FBt1/assignment_loss‚0.003051587118824462 1loss/SRqqt2/detection_loss‚0.6324496404755091 _runtime‚8111.982315281 2loss/SRqqt2/assignment_loss‚0.4941008450204032 2loss/FBt2/assignment_loss‚0.017141616021157456 %loss/total_loss‚7.800400821381927 epoch‚22 % _timestamp‚1.7845149384069216e+09 _step‚737 0loss/FRt1/assignment_loss‚1.6574668686004426 /loss/FRt1/detection_loss‚0.4897961386257354 /loss/FRt2/detection_loss‚1.4416201000004534 1loss/SRqqt1/detection_loss‚0.7555876877634045í ã©œ‰ 2loss/FBt2/assignment_loss‚0.017141616021157456 %loss/total_loss‚7.800400821381927 epoch‚22 % _timestamp‚1.7845149384069216e+09 _step‚737 0loss/FRt1/assignment_loss‚1.6574668686004426 /loss/FRt1/detection_loss‚0.4897961386257354 /loss/FRt2/detection_loss‚1.4416201000004534 1loss/SRqqt1/detection_loss‚0.7555876877634045 0loss/FRt2/assignment_loss‚0.7146945302603674 /loss/FBt1/detection_loss‚0.7971126196874927 /loss/FBt2/detection_loss‚0.3574364054665873 trainer/global_step‚8949 2loss/SRqqt1/assignment_loss‚0.4399427823415489 2loss/FBt1/assignment_loss‚0.003051587118824462 1loss/SRqqt2/detection_loss‚0.6324496404755091 _runtime‚8111.982315281 2loss/SRqqt2/assignment_loss‚0.4941008450204032á‹b‚(:× ý’öÒÍœÇË network.recv‚ 2.167775e+06( disk./.usagePercent‚75.9893355550488( disk.nvme1c1n1.out‚24792.67724609375 disk.nvme1n1.in‚0 disk.md0.in‚0# disk.dm-0.out‚59406.25732421875 disk.dm-1.in‚0$ memory_percent‚96.78609371185303% disk./.usageGB‚1166.6262741088867 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme0n1.out‚46905.5751953125& disk.nvme1n1.out‚24792.67724609375 disk.dm-0.in‚ 0.00390625( proc.memory.availableMB‚ 526.56640625# proc.memory.rssMB‚ 3020.71484375) proc.memory.percent‚18.43698024749756 network.sent‚ 1.4284538e+07 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0 disk.dm-1.out‚ 11772.6328125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚46905.5751953125 disk.md0.out‚ 242.828125 disk.rbd0.out‚0 cpu‚25.693602040561714 proc.cpu.threads‚17‘xë…žž‚(:“ ý’öÒ΢…Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚233.281' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.76033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`…¶Õ^Ÿ‚(:Ó Œ“öÒ×å¼Ë network.sent‚ 1.433594e+07) disk./.usagePercent‚75.98935195433357 disk.nvme0n1.in‚ 0.00390625 disk.dm-0.in‚ 0.00390625$ memory_percent‚96.78606986999512) proc.memory.percent‚18.43698024749756 network.recv‚ 2.171984e+06 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚46906.25927734375 disk.nvme1n1.in‚0' proc.memory.availableMB‚ 526.5703125# proc.memory.rssMB‚ 3020.71484375 cpu‚25.693184767992275( disk.nvme0c0n1.out‚46906.25927734375$ disk.nvme1n1.out‚24793.048828125 disk.md0.in‚0 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚24793.048828125% disk./.usageGB‚1166.6265258789062 disk.dm-1.in‚0 disk.dm-1.out‚ 11772.7109375 disk.rbd0.out‚0 disk.md0.out‚ 243.03125# disk.dm-0.out‚59406.7963867187509õc ‚(:ˆ Œ“öÒäƆÌ gpu.0.gpu‚79.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.23·dBê09273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.756' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øí6_¡‚(:Ô ›“öÒüðÅË disk.rbd0.out‚0 disk.nvme1c1n1.in‚0$ memory_percent‚96.78628444671631 proc.cpu.threads‚17# disk.dm-1.out‚11772.82763671875& disk.nvme0n1.out‚46908.38427734375# proc.memory.rssMB‚ 3020.71484375) proc.memory.percent‚18.43698024749756) disk./.usagePercent‚75.98936065092398 disk.nvme1n1.in‚0! disk.nvme1n1.out‚ 24793.671875 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0# disk.nvme1c1n1.out‚ 24793.671875( proc.memory.availableMB‚ 526.53515625 network.sent‚ 1.4340775e+07 network.recv‚ 2.175525e+06 disk.md0.in‚0 disk.md0.out‚ 243.2890625 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚46908.38427734375 disk.nvme0n1.in‚ 0.00390625 cpu‚25.692829179937466% disk./.usageGB‚1166.6266593933105" disk.dm-0.out‚59408.8447265625 disk.dm-1.in‚0P”+‘¢‚(:† ›“öÒù±Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚260.631' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚86.877 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾SƒL2£- _wandbruntime‚8155  _runtime‚8155ÌÙô¥É¤Ã % _timestamp‚1.7845149817526267e+09 _runtime‚8155.329038759 _step‚738 ' lr-AdamW/pg1‚0.00036602552647238177 ' lr-AdamW/pg2‚0.00036602552647238177 trainer/global_step‚8999¦vÎ¥È ' lr-AdamW/pg2‚0.00036602552647238177 trainer/global_step‚8999 % _timestamp‚1.7845149817526267e+09 _runtime‚8155.329038759 _step‚738 ' lr-AdamW/pg1‚0.00036602552647238177â"Ê(“2¦- _wandbruntime‚8155  _runtime‚8155éAæ1(§ûO _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10434743060918 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10450170398208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚309756870103552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37201218 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1387688188468224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10509460226048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _runtime‚8155.329107769 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72842679 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10450170398208 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10450511822336 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10450511822336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1377237659606528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89027517 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚309756746222080 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1387688188468224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1379613588023788 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1387687830004736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161880341 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10509354956800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1377237676645888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21499339 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1369179202563504 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15702165 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1369179185524144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21499053 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚299247391265280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89027517 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1377237676645888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89037660 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 % _timestamp‚1.7845149817533736e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10434402499644 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1377237659606528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72842681 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72842681 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37201507 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15702168 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚739 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1387687830004736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72842679 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89037660 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161870196 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 trainer/global_step‚8999žqÌ(¨€P L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72842679 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10450170398208 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10450511822336 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10450511822336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1377237659606528 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89027517 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚309756746222080 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1387688188468224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1379613588023788 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1387687830004736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161880341 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10509354956800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1377237676645888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21499339 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1369179202563504 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15702165 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1369179185524144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21499053 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚299247391265280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89027517 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1377237676645888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89037660 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 % _timestamp‚1.7845149817533736e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10434402499644 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1377237659606528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72842681 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72842681 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37201507 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15702168 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚739 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1387687830004736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72842679 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89037660 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161870196 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 trainer/global_step‚8999 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10434743060918 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10450170398208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚309756870103552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37201218 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1387688188468224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10509460226048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _runtime‚8155.329107769ãÇf¥(2©- _wandbruntime‚8155  _runtime‚8155‹þÓ'ª‚N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299280571535360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161887645 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89037173 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10451280497152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1377385934108672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161887645 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚309791048640512 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72850474 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89047330 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 # _timestamp‚1.78451498237942e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1377385951148032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1369326633230976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚740 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161897804 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72850474 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89037173 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10451621928448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10510573978624 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1379762127079512 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚8155.955217569 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1387837573076480 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1387837573076480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1377385934108672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299280590147584 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10435851449282 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚8999 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10451621928448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37205547 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1369326616191616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21501356 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89047330 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161897804 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37205255 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1377385951148032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15703899 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10451280497152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1377385951148032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1369326633230976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚740 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161897804 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72850474 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89037173 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10451621928448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10510573978624 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1379762127079512 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚8155.955217569 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1387837573076480 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1387837573076480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1377385934108672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299280590147584 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10435851449282 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚8999 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10451621928448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37205547 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1369326616191616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21501356 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89047330 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161897804 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37205255 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1377385951148032 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15703899 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10451280497152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10451280497152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1377385934108672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161887645 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚309791048640512 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180ä[ÿ÷2¬- _wandbruntime‚8155  _runtime‚8155‹SI% ­… 0loss/FRt2/assignment_loss‚0.7722086783061896 2loss/SRqqt1/assignment_loss‚0.5098881030661401 3loss/SRqqt2/assignment_loss‚0.46217048288281226 /loss/FRt2/detection_loss‚1.6409213662825692 1loss/SRqqt2/detection_loss‚0.6870098000919437 /loss/FBt1/detection_loss‚0.4161655982200106 % _timestamp‚1.7845149823805144e+09 _runtime‚8155.955844569 2loss/FBt1/assignment_loss‚0.003667985784031089 %loss/total_loss‚7.788913590428369 epoch‚22 1loss/FBt2/assignment_loss‚0.02267352619106938 0loss/FRt1/detection_loss‚0.47690768910531983 1loss/SRqqt1/detection_loss‚0.6234900549966462 /loss/FBt2/detection_loss‚0.5585243965974672 trainer/global_step‚8999 _step‚741 0loss/FRt1/assignment_loss‚1.6152859089041707—ßu®Š 1loss/FBt2/assignment_loss‚0.02267352619106938 0loss/FRt1/detection_loss‚0.47690768910531983 1loss/SRqqt1/detection_loss‚0.6234900549966462 /loss/FBt2/detection_loss‚0.5585243965974672 trainer/global_step‚8999 _step‚741 0loss/FRt1/assignment_loss‚1.6152859089041707 0loss/FRt2/assignment_loss‚0.7722086783061896 2loss/SRqqt1/assignment_loss‚0.5098881030661401 3loss/SRqqt2/assignment_loss‚0.46217048288281226 /loss/FRt2/detection_loss‚1.6409213662825692 1loss/SRqqt2/detection_loss‚0.6870098000919437 /loss/FBt1/detection_loss‚0.4161655982200106 % _timestamp‚1.7845149823805144e+09 _runtime‚8155.955844569 2loss/FBt1/assignment_loss‚0.003667985784031089 %loss/total_loss‚7.788913590428369 epoch‚22åI*¥dc¯‚(:Ø ª“öÒˆÜ³Ë disk.dm-0.in‚ 0.00390625 network.sent‚ 1.4343697e+07) disk./.usagePercent‚75.98934201537311 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚46909.00439453125$ memory_percent‚96.78645133972168) proc.memory.percent‚18.43698024749756 network.recv‚ 2.178705e+06% disk./.usageGB‚1166.6263732910156 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.md0.in‚0 disk.md0.out‚243.4375# proc.memory.rssMB‚ 3020.71484375 cpu‚25.692384629178942& disk.nvme0n1.out‚46909.00439453125 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚24794.0146484375" disk.dm-0.out‚59409.4228515625' proc.memory.availableMB‚ 526.5078125 proc.cpu.threads‚17# disk.dm-1.out‚11772.87060546875 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme1c1n1.out‚24794.0146484375 disk.nvme1n1.in‚0¦ ¾“°‚(:ˆ ª“öÒ´áË gpu.0.gpu‚75.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.961' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.987 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓÈ8ê[±‚(:Ð ¹“öÒþ•ÁË disk.md0.in‚0( disk.nvme1c1n1.out‚24794.68798828125 disk.nvme1n1.in‚0$ memory_percent‚96.78797721862793 proc.cpu.threads‚17) disk./.usagePercent‚75.98934077300305 disk.dm-0.in‚ 0.00390625! disk.dm-1.out‚11773.005859375 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚46910.53369140625# proc.memory.rssMB‚ 3020.71484375 disk.md0.out‚ 243.71875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚46910.53369140625& disk.nvme1n1.out‚24794.68798828125) proc.memory.percent‚18.43698024749756 network.sent‚ 1.440628e+07 network.recv‚ 2.182842e+06 disk.dm-0.out‚ 59410.875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1c1n1.in‚0' proc.memory.availableMB‚ 526.2578125 cpu‚25.69194007148644% disk./.usageGB‚1166.6263542175293ŸæHž²‚(:“ ¹“öÒã’Ì gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚270.213' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.07100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;séØ_³‚(:Ô È“öÒÒÜÀË disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚46911.9033203125 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0! disk.dm-1.out‚11773.056640625 network.sent‚ 1.4411987e+07 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚24795.5849609375 disk.nvme0c0n1.in‚ 0.00390625' disk.nvme0c0n1.out‚46911.9033203125% disk./.usageGB‚1166.6269264221191$ memory_percent‚96.78800106048584( proc.memory.availableMB‚ 526.25390625# proc.memory.rssMB‚ 3020.71484375) proc.memory.percent‚18.43698024749756 cpu‚25.69155878642355 network.recv‚ 2.185552e+06) disk./.usagePercent‚75.98937804410481' disk.nvme1c1n1.out‚24795.5849609375 disk.md0.in‚0 disk.md0.out‚ 244.4453125 disk.rbd0.out‚0 disk.nvme1n1.in‚0 disk.dm-0.out‚ 59411.5859375pòÚø“´‚(:ˆ È“öÒ´üË gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚269.673' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚89.891 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úYP2µ- _wandbruntime‚8198  _runtime‚8198>¿<7ƶÀ _runtime‚8198.885023398 _step‚742 & lr-AdamW/pg1‚0.0003636068066369269 & lr-AdamW/pg2‚0.0003636068066369269 trainer/global_step‚9049 $ _timestamp‚1.784515025308536e+09ð@“Ë·Å trainer/global_step‚9049 $ _timestamp‚1.784515025308536e+09 _runtime‚8198.885023398 _step‚742 & lr-AdamW/pg1‚0.0003636068066369269 & lr-AdamW/pg2‚0.0003636068066369269æx­l2¸- _wandbruntime‚8198  _runtime‚8198’¾¿wÿ'¹ùO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162744498 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89512169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1395157482677248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1395157124213760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10506080924160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10490226274326 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10489885713052 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10506080924160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1384651384713728 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15788865 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1395157482677248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89522312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15788868 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162754643 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300906423381504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10505739500032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73232329 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1387040962211430 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1384651384713728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1376550718897744 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚743 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚8198.885083008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37402681 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10565259764736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37402969 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311471557167616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89522312 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300906404769280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73232331 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10505739500032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176ÄÊ豺þO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1376550735937104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21613816 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1384651384713728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1376550718897744 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚743 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚8198.885083008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37402681 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10565259764736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37402969 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311471557167616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89522312 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300906404769280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73232331 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10505739500032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162744498 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89512169 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1395157482677248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1395157124213760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10506080924160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10490226274326 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10489885713052 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10506080924160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1384651384713728 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15788865 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1395157482677248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceÂo΀SStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89522312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15788868 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162754643 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300906423381504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10505739500032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73232329 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1387040962211430 J; '¼„N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162761947 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89531982 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311505977310208 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73240124 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37406691 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73240122 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10566373658624 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1387189143666520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10507191030272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10506849598976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37406976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311505859727360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1376698166604576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89521825 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1384799676255232 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1395306508814848 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162772106 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10506849598976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚9049 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89531982 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21616374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15790602 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162772106 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1384799659215872 _runtime‚8199.510752166 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1387189501267266 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1384799676255232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1395306867285504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10566274688000ý{QY'½‰N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73240124 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311505977310208 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73240124 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37406691 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73240122 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10566373658624 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1387189143666520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10507191030272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10506849598976 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37406976 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311505859727360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1376698166604576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89521825 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1384799676255232 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1395306508814848 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162772106 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10506849598976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 trainer/global_step‚9049 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89531982 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21616374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15790602 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162772106 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1384799659215872 _runtime‚8199.510752166 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1387189501267266 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1384799676255232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1395306867285504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10566274688000 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162761947 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89531982 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚9099 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90006964 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1392065126860288 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1402626776886272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1394467621197804 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1402626776886272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89996821 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10561650025984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15875565 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚302565436885504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1383922252271344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73621981 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163628945 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1402626418422784 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1392065109820928 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10561308601856 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1392065126860288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚302565418273280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21728418 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1394467978798438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90006964 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1402626418422784 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚9099 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90006964 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1392065126860288 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1402626776886272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1394467621197804 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1402626776886272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89996821 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10561650025984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15875565 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚302565436885504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1383922252271344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73621981 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163628945 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1402626418422784 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1392065109820928 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10561308601856 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1392065126860288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚302565418273280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21728418 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1394467978798438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90006964 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1402626418422784 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37607973 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1392213401362432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21730674 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10546477314712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1394616517854274 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10546817876098 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10562760132096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚297 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163646408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚313220721711104 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1384069682938816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163646408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21730971 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _step‚748 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37608273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15877302 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1402776161494528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚302598598543360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73629774 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90016634 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90006477 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10562418700800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 $ _timestamp‚1.784515069908746e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1402776161494528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10622123167744 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73629774 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1392213384323072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10622232624128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163636249 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163646408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21730971 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _step‚748 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37608273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15877302 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1402776161494528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚302598598543360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73629774 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90016634 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90006477 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10562418700800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 $ _timestamp‚1.784515069908746e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1402776161494528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10622123167744 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73629774 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1392213384323072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10622232624128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163636249 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37607973 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1392213401362432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21730674 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10546477314712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1394616517854274 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10546817876098 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10562760132096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚297 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163646408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚313220721711104 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1384069682938816ì¤x«2Ð- _wandbruntime‚8243  _runtime‚8243~3—8 Ñ„ 0loss/FRt2/assignment_loss‚0.7298784655247008 1loss/FBt2/assignment_loss‚0.03141528901586412 .loss/FRt2/detection_loss‚1.533216727353882 1loss/SRqqt1/detection_loss‚0.6165047653369827 0loss/FBt1/detection_loss‚0.47675420211749225 $ _timestamp‚1.784515069909789e+09 2loss/SRqqt1/assignment_loss‚0.5301542203531686 /loss/FBt2/detection_loss‚0.3444849461695905 2loss/SRqqt2/assignment_loss‚0.6023773280489819 3loss/FBt1/assignment_loss‚0.0017146944431165204 0loss/FRt1/detection_loss‚0.45560134161317517 %loss/total_loss‚7.744117862159387 epoch‚22 _runtime‚8243.485208923 0loss/FRt1/assignment_loss‚1.7645789457611802 1loss/SRqqt2/detection_loss‚0.6574369364212527 trainer/global_step‚9099 _step‚749ôÖ,Ò‰ 2loss/SRqqt2/assignment_loss‚0.6023773280489819 3loss/FBt1/assignment_loss‚0.0017146944431165204 0loss/FRt1/detection_loss‚0.45560134161317517 %loss/total_loss‚7.744117862159387 epoch‚22 _runtime‚8243.485208923 0loss/FRt1/assignment_loss‚1.7645789457611802 1loss/SRqqt2/detection_loss‚0.6574369364212527 trainer/global_step‚9099 _step‚749 0loss/FRt2/assignment_loss‚0.7298784655247008 1loss/FBt2/assignment_loss‚0.03141528901586412 .loss/FRt2/detection_loss‚1.533216727353882 1loss/SRqqt1/detection_loss‚0.6165047653369827 0loss/FBt1/detection_loss‚0.47675420211749225 $ _timestamp‚1.784515069909789e+09 2loss/SRqqt1/assignment_loss‚0.5301542203531686 /loss/FBt2/detection_loss‚0.3444849461695905í£{* žÓ‚(:“ „”öÒìþ¨Ë gpu.0.gpu‚46.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚288.049' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.01633333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À/_dÔ‚(:Ù „”öÒ¿ìÁË$ disk./.usageGB‚1166.627212524414( disk.nvme0c0n1.out‚46917.77197265625& disk.nvme0n1.out‚46917.77197265625( disk.nvme1c1n1.out‚24798.09619140625 disk.nvme1n1.in‚0 network.sent‚ 1.4488197e+07 disk.md0.out‚ 245.765625 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚11773.4990234375" disk.dm-0.out‚59416.6513671875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625$ memory_percent‚96.79036140441895# proc.memory.rssMB‚ 3020.71484375 cpu‚25.689926356449078 proc.cpu.threads‚17 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚24798.09619140625' proc.memory.availableMB‚ 525.8671875) proc.memory.percent‚18.43698024749756 network.recv‚ 2.199113e+06( disk./.usagePercent‚75.9893966796557ζ]veÕ‚(:Ú “”öÒ¤Á¬Ë) proc.memory.percent‚18.43698024749756 cpu‚25.689523049125558 disk.nvme0c0n1.in‚ 0.00390625% disk./.usageGB‚1166.6279640197754 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚59419.26708984375 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme1n1.out‚24798.91552734375 disk.md0.out‚ 246.203125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚46920.5830078125( disk.nvme1c1n1.out‚24798.91552734375( proc.memory.availableMB‚ 526.36328125# proc.memory.rssMB‚ 3020.71484375 proc.cpu.threads‚17 network.sent‚ 1.4549378e+07) disk./.usagePercent‚75.98944562903601 network.recv‚ 2.203231e+06! disk.dm-1.out‚11773.576171875' disk.nvme0c0n1.out‚46920.5830078125 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0$ memory_percent‚96.78733348846436›­ñ_Ö‚(:’ “”öÒœ³‰Ì gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.892' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.29733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž3cׂ(:Ø ¢”öÒ“ê¢Ë disk.md0.out‚ 246.46875# proc.memory.rssMB‚ 3020.71484375) proc.memory.percent‚18.43698024749756 network.sent‚ 1.4556223e+07 disk.nvme0c0n1.in‚ 0.00390625& disk.nvme0n1.out‚46922.63916015625 disk.md0.in‚0$ memory_percent‚96.78733348846436 cpu‚25.6890603292296 proc.cpu.threads‚17 network.recv‚ 2.206557e+06) disk./.usagePercent‚75.98954700643282 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 526.36328125 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚46922.63916015625 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚24799.51904296875 disk.dm-0.in‚ 0.00390625" disk.dm-0.out‚59421.2626953125% disk./.usageGB‚1166.6295204162598! disk.dm-1.out‚11773.650390625 disk.nvme1n1.in‚0& disk.nvme1n1.out‚24799.51904296875(•öTØ‚(:’ ¢”öÒÏõéË gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚271.98' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.66000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A€™E2Ù- _wandbruntime‚8286  _runtime‚82866uÕÅÚ¿ trainer/global_step‚9149 # _timestamp‚1.78451511314095e+09 _runtime‚8286.716480254 _step‚750 & lr-AdamW/pg1‚0.0003587586624521238 & lr-AdamW/pg2‚0.0003587586624521238˜e¢AÊÛÄ # _timestamp‚1.78451511314095e+09 _runtime‚8286.716480254 _step‚750 & lr-AdamW/pg1‚0.0003587586624521238 & lr-AdamW/pg2‚0.0003587586624521238 trainer/global_step‚9149îÝ2š2Ü- _wandbruntime‚8286  _runtime‚8286`Ê z(ÝúO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37805219 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21842954 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1401894637784812 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1410095712631808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1399478851967488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3ñzp|(ÞÿO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37805219 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21842954 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1401894637784812 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1410095712631808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1399478851967488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164520710 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74019422 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37809160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1410245455703552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1410245097232896 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10617987802624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10677958881792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10618329233920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164520710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15963999 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚8287.341798746 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90491129 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37809450 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10618329233920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164510551 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10678068338176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚9149 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90501286 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚314935570929152 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1402043176840536 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1410245455703552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10602301089506 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10617987802624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90501286 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21845161 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304257612047360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1399627109430272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164510551 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1399627109430272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74019422ù‘4"'áˆN 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10618329233920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164510551 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10678068338176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚9149 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90501286 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚314935570929152 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1402043176840536 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1410245455703552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10602301089506 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10617987802624 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90501286 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21845161 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚304257612047360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1399627109430272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164510551 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1399627109430272 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74019422 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1402043534441282 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21845448 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1391441233351776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74019424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164520710 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74019422 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37809160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1410245455703552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1410245097232896 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10617987802624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10677958881792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10618329233920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164520710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15963999 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚8287.341798746 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90491129 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37809450ð=ó6Î2â- _wandbruntime‚8287  _runtime‚8287 çÿ ãƒ %loss/total_loss‚8.372487700554357 0loss/FRt2/assignment_loss‚0.8605540765381847 2loss/SRqqt1/assignment_loss‚0.4797912213732118 1loss/FBt1/assignment_loss‚0.00910507686902134 /loss/FRt1/detection_loss‚0.4899549244666181 1loss/SRqqt2/detection_loss‚0.7443526653430016 /loss/FBt2/detection_loss‚0.8085793540114536 0loss/FRt1/assignment_loss‚1.6481657392417814 1loss/SRqqt1/detection_loss‚0.6243737633860382 /loss/FBt1/detection_loss‚0.5220177453802275 % _timestamp‚1.7845151137678726e+09 _step‚753 2loss/SRqqt2/assignment_loss‚0.5126472939813782 /loss/FRt2/detection_loss‚1.6369146034218023 _runtime‚8287.343271749 2loss/FBt2/assignment_loss‚0.036031236541637666 epoch‚22 trainer/global_step‚9149k>´…äˆ %loss/total_loss‚8.372487700554357 0loss/FRt2/assignment_loss‚0.8605540765381847 2loss/SRqqt1/assignment_loss‚0.4797912213732118 1loss/FBt1/assignment_loss‚0.00910507686902134 /loss/FRt1/detection_loss‚0.4899549244666181 1loss/SRqqt2/detection_loss‚0.7443526653430016 /loss/FBt2/detection_loss‚0.8085793540114536 0loss/FRt1/assignment_loss‚1.6481657392417814 1loss/SRqqt1/detection_loss‚0.6243737633860382 /loss/FBt1/detection_loss‚0.5220177453802275 % _timestamp‚1.7845151137678726e+09 _step‚753 2loss/SRqqt2/assignment_loss‚0.5126472939813782 /loss/FRt2/detection_loss‚1.6369146034218023 _runtime‚8287.343271749 2loss/FBt2/assignment_loss‚0.036031236541637666 epoch‚22 trainer/global_step‚9149ñoçhå‚(:Ý ±”öÒ–—îË) disk./.usagePercent‚75.98957160535997& disk.nvme1n1.out‚24800.34716796875 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚96.78905010223389( proc.memory.availableMB‚ 526.08203125 network.recv‚ 2.210055e+06$ disk./.usageGB‚1166.629898071289 disk.nvme1c1n1.in‚0 disk.md0.out‚ 246.70703125" disk.dm-0.out‚59424.4833984375( disk.nvme0c0n1.out‚46925.72412109375# proc.memory.rssMB‚ 3020.71484375) proc.memory.percent‚18.43698024749756 network.sent‚ 1.4571983e+07 disk.dm-0.in‚ 0.00390625" disk.dm-1.out‚11773.8017578125 cpu‚25.689166501104904 proc.cpu.threads‚17( disk.nvme1c1n1.out‚24800.34716796875 disk.nvme1n1.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme0n1.in‚ 0.00390625& disk.nvme0n1.out‚46925.72412109375ÿÓA æ‚(:’ ±”öÒ¤ú®Ì gpu.0.gpu‚14.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚276.377' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.12566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A õjdç‚(:Ù À”öÒ¥Ë´Ë network.recv‚ 2.214133e+06& disk.nvme1n1.out‚24801.09716796875 disk.dm-0.in‚ 0.00390625 disk.nvme0c0n1.in‚ 0.00390625( disk.nvme0c0n1.out‚46927.21728515625# proc.memory.rssMB‚ 3020.71484375 disk.nvme1n1.in‚0 disk.md0.in‚0" disk.dm-1.out‚11773.8447265625 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme1c1n1.out‚24801.09716796875& proc.memory.availableMB‚ 526.296875) disk./.usagePercent‚75.98958179279445$ disk./.usageGB‚1166.630054473877 disk.md0.out‚ 247.30078125 disk.dm-1.in‚0& disk.nvme0n1.out‚46927.21728515625) proc.memory.percent‚18.43698024749756 proc.cpu.threads‚17 disk.dm-0.out‚59425.43359375 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0$ memory_percent‚96.78773880004883 cpu‚25.689729385743235 network.sent‚ 1.4623333e+07ñò Ðè‚(:’ À”öÒ¸¶ðË gpu.0.gpu‚91.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚264.07' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.02333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªUëbé‚(:× Ï”öÒÐÄÐË' disk.nvme0c0n1.out‚46929.3154296875 disk.nvme0n1.in‚ 0.00390625% disk.nvme0n1.out‚46929.3154296875$ memory_percent‚96.78776264190674) disk./.usagePercent‚75.98952911630397% disk./.usageGB‚1166.6292457580566% disk.nvme1c1n1.out‚24801.91796875 disk.nvme1n1.in‚0# disk.nvme1n1.out‚24801.91796875 disk.dm-1.in‚0# disk.dm-1.out‚11773.94580078125 disk.nvme0c0n1.in‚ 0.00390625 network.recv‚ 2.217885e+06 disk.md0.in‚0 disk.md0.out‚ 247.8046875 network.sent‚ 1.4628184e+07 disk.nvme1c1n1.in‚0( proc.memory.availableMB‚ 526.29296875# proc.memory.rssMB‚ 3020.71484375) proc.memory.percent‚18.43698024749756 cpu‚25.690255965860867 proc.cpu.threads‚17 disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚59427.18701171875 disk.rbd0.in‚0 disk.rbd0.out‚0(JØÌ“ê‚(:ˆ Ï”öÒ¿Ê‘Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚124.887' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚41.629 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©Þ[42ë- _wandbruntime‚8330  _runtime‚8330DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1417565006840832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚305883445281280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21957927 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚8330.361362493 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚316616266810880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10732802917376 _step‚755 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74401281 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74401279 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1417565365304320 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1406892560035328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10656675914550 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90976268 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1406892560035328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74401279 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1406892577074688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38006608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165367404 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1398665319018544 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1406892577074688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚9199 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1417565365304320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21957643 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1417565006840832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10672446805504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16048968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90966125 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1409322011972454 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10672788229632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128ß&OV(ð€P TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21957927 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚8330.361362493 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚316616266810880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10732802917376 _step‚755 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74401281 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74401279 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1417565365304320 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1406892560035328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10656675914550 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90976268 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1406892560035328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74401279 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1406892577074688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38006608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165367404 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1398665319018544 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1406892577074688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚9199 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1417565365304320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21957643 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1417565006840832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10672446805504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16048968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90966125 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1409322011972454 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10672788229632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10672446805504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16048965 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10656335353276 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 % _timestamp‚1.7845151567858248e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1417565006840832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚305883445281280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272ó­7ÂÈ2ñ- _wandbruntime‚8330  _runtime‚8330`ˆ´É 'ò…N ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10733807467520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1409470193427544 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16050699 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10657784302914 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165384853 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38010636 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10657443741528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1407040851576832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316650433018880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21960234 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165395012 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90975781 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1417714749912576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1417714749912576 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316650556893184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10673898335744 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 trainer/global_step‚9199 _step‚756 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74409072 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90985938 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10673898335744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74409074 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1407040834537472 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10673556904448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336X†´'óŠN UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1417714749912576 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316650556893184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10673898335744 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 trainer/global_step‚9199 _step‚756 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74409072 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90985938 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10673898335744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74409074 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1407040834537472 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10673556904448 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10733807467520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1409470193427544 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16050699 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10657784302914 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165384853 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38010636 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10657443741528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1407040851576832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316650433018880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21960234 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165395012 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90975781 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1417714749912576ô1®i2ô- _wandbruntime‚8330  _runtime‚83303çäÑõƒ 0loss/FRt2/assignment_loss‚0.9047681363444254 2loss/FBt2/assignment_loss‚0.013525700462750897 /loss/FRt1/detection_loss‚0.4760374532745411 /loss/FBt2/detection_loss‚0.7709328363211277 # éÖ8loss/total_loss‚8.4978379210006 _step‚757 0loss/FRt1/assignment_loss‚1.7251201764229804 2loss/SRqqt2/assignment_loss‚0.5528728539473486 3loss/FBt1/assignment_loss‚0.0034334490559006425 /loss/FRt2/detection_loss‚1.7715807954920697 1loss/SRqqt1/detection_loss‚0.5735163824919906 1loss/SRqqt2/detection_loss‚0.7169753936608831 epoch‚22 2loss/SRqqt1/assignment_loss‚0.5431376402912452 trainer/global_step‚9199 % _timestamp‚1.7845151574061654e+09 _runtime‚8330.981481227 /loss/FBt1/detection_loss‚0.4459371032353374slãöˆ /loss/FBt1/detection_loss‚0.4459371032353374 0loss/FRt2/assignment_loss‚0.9047681363444254 2loss/FBt2/assignment_loss‚0.013525700462750897 /loss/FRt1/detection_loss‚0.4760374532745411 /loss/FBt2/detection_loss‚0.7709328363211277 #loss/total_loss‚8.4978379210006 _step‚757 0loss/FRt1/assignment_loss‚1.7251201764229804 2loss/SRqqt2/assignment_loss‚0.5528728539473486 3loss/FBt1/assignment_loss‚0.0034334490559006425 /loss/FRt2/detection_loss‚1.7715807954920697 1loss/SRqqt1/detection_loss‚0.5735163824919906 1loss/SRqqt2/detection_loss‚0.7169753936608831 epoch‚22 2loss/SRqqt1/assignment_loss‚0.5431376402912452 trainer/global_step‚9199 % _timestamp‚1.7845151574061654e+09 _runtime‚8330.981481227õعè6b÷‚(:× Þ”öÒÇœÄË disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme1c1n1.out‚24802.55859375 disk.md0.out‚ 248.17578125 network.recv‚ 2.220922e+06 disk.dm-0.in‚ 0.00390625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1n1.in‚0# disk.nvme1n1.out‚24802.55859375$ memory_percent‚96.77834510803223' proc.memory.availableMB‚ 527.8359375) disk./.usagePercent‚75.98953483120624# disk.dm-1.out‚11774.08251953125' disk.nvme0c0n1.out‚46930.1865234375% disk.nvme0n1.out‚46930.1865234375 disk.nvme1c1n1.in‚0# proc.memory.rssMB‚ 3020.71484375 disk.md0.in‚0) proc.memory.percent‚18.43698024749756 cpu‚25.693385528371657 proc.cpu.threads‚17 network.sent‚ 1.4643949e+07% disk./.usageGB‚1166.6293334960938# disk.dm-0.out‚59427.74951171875U÷w¸žø‚(:“ Þ”öÒÕ°‚Ì gpu.0.gpu‚97.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.557' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.18566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ƒÜ2ù- _wandbruntime‚8348  _runtime‚8348dì”@=úŠ{ 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9917741417884827 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5167285203933716 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6428035497665405 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23970234394073486 *jet/accuracy_1_of_1‚0.2825782299041748 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5964245796203613 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 particle/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21895238757133484 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8947467803955078 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2690601050853729 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2001316249370575 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1550515741109848 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6628110408782959 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6392574310302734 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7265625596046448 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49632981419563293 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5698865652084351 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21547885239124298 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.9056500196456909 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5052083134651184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19868247210979462 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24293673038482666 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.27744895219802856 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6026147603988647 +jet/accuracy_3_of_4‚0.09097591042518616 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5442656874656677 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16784504055976868 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6676167249679565 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12906844913959503 +particle/specificity‚0.9072546362876892 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21279339492321014 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.575833261013031 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958332777023315 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2469220608472824 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19868247210979462 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974085688591003 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5418379306793213 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5665336847305298 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6882026195526123 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16980794072151184 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12906844913959503 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997290849685669 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5089985728263855 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21289542317390442 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9077475666999817 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9881629943847656 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2299463152885437 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2021147906780243 trainer/global_step‚9199 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23022691905498505 jet/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9156157374382019 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9899949431419373 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.669242262840271 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 _runtime‚8348.264670371 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6884473562240601 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21612626314163208 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21517635881900787 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9042706489562988 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45442554354667664 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15618422627449036 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5764763951301575 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 *jet/accuracy_2_of_4‚0.7587796449661255 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2051434963941574 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43981480598449707 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6830708980560303 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_4‚0.9965986609458923 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21786759793758392 particle/accuracy_2_of_4‚1 _step‚758 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.651515007019043 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 epoch‚22 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9894712567329407 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2877899408340454 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06318183988332748 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6204833984375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.26622438430786133 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6185691356658936 +jet/accuracy_4_of_4‚0.01670061983168125 (particle/accuracy‚0.8493186235427856 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03976878523826599 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.032696835696697235 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_2_of_2‚0.13769657909870148 particle/accuracy_1_of_3‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.30593037605285645 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9949482679367065 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_1_of_5‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9116351008415222 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6187693476676941 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6821587085723877 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 )loss/val_total_loss‚7.954708618942478 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981710314750671 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7521836161613464 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884457588195801 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32607683539390564 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.9077475666999817 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7587581276893616 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8944104313850403 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28215622901916504 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9223443269729614 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5396057367324829 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6249738931655884 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6334924697875977 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12127397954463959 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15296179056167603 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5302351117134094 /particle/accuracy_4_of_4‚0.9498055577278137 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7593507170677185 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1918960064649582 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968745708465576 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9799898266792297 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.624453067779541 7validation_average_jet_accuracy‚0.33279159665107727 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32016879320144653 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5824978947639465 particle/accuracy_1_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7079201936721802 'particle/f_score‚0.6450176239013672 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6292538046836853 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949500560760498 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.050447214394807816 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912253022193909 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5730595588684082 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5852103233337402 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5839623212814331 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_3‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12367166578769684 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5366381406784058 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5357201099395752 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919624328613281 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16530713438987732 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.31573235988616943 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6223000884056091 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8707223534584045 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5357201099395752 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46064817905426025 +jet/accuracy_3_of_3‚0.07664580643177032 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2410530298948288 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 particle/accuracy_3_of_5‚1 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.996280312538147 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5760143399238586 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6113748550415039 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_2_of_3‚1 +particle/sensitivity‚0.6375042796134949 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04153487831354141 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7521836161613464 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305934071540833 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23069792985916138 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.021440740674734116 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9881424903869629 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49778419733047485 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7082000970840454 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 $ _timestamp‚1.784515174688627e+09 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9573813080787659å‡ø…Ç7û{ & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.032696835696697235 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_2_of_2‚0.13769657909870148 particle/accuracy_1_of_3‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.30593037605285645 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9949482679367065 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_1_of_5‚1 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9116351008415222 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6187693476676941 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6821587085723877 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 )loss/val_total_loss‚7.954708618942478 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981710314750671 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7521836161613464 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9884457588195801 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32607683539390564 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.9077475666999817 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7587581276893616 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8944104313850403 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28215622901916504 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9223443269729614 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5396057367324829 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6249738931655884 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6334924697875977 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12127397954463959 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15296179056167603 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5302351117134094 /particle/accuracy_4_of_4‚0.9498055577278137 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7593507170677185 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1918960064649582 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9968745708465576 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9799898266792297 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.624453067779541 7validation_average_jet_accuracy‚0.33279159665107727 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32016879320144653 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5824978947639465 particle/accuracy_1_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7079201936721802 'particle/f_score‚0.6450176239013672 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6292538046836853 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949500560760498 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.050447214394807816 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912253022193909 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5730595588684082 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5852103233337402 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5839623212814331 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_3‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12367166578769684 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5366381406784058 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5357201099395752 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919624328613281 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16530713438987732 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.31573235988616943 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6223000884056091 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8707223534584045 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5357201099395752 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46064817905426025 +jet/accuracy_3_of_3‚0.07664580643177032 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2410530298948288 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 particle/accuracy_3_of_5‚1 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.996280312538147 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5760143399238586 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6113748550415039 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_2_of_3‚1 +particle/sensitivity‚0.6375042796134949 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04153487831354141 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7521836161613464 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8305934071540833 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23069792985916138 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.021440740674734116 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9881424903869629 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49778419733047485 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7082000970840454 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 $ _timestamp‚1.784515174688627e+09 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9573813080787659 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9917741417884827 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5167285203933716 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6428035497665405 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23970234394073486 *jet/accuracy_1_of_1‚0.2825782299041748 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5964245796203613 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 particle/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21895238757133484 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8947467803955078 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2690601050853729 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2001316249370575 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1550515741109848 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6628110408782959 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6392574310302734 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7265625596046448 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49632981419563293 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5698865652084351 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21547885239124298 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.9056500196456909 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5052083134651184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19868247210979462 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24293673038482666 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.27744895219802856 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6026147603988647 +jet/accuracy_3_of_4‚0.09097591042518616 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5442656874656677 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16784504055976868 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6676167249679565 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12906844913959503 +particle/specificity‚0.9072546362876892 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21279339492321014 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.575833261013031 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958332777023315 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2469220608472824 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19868247210979462 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974085688591003 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5418379306793213 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5665336847305298 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6882026195526123 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16980794072151184 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12906844913959503 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997290849685669 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5089985728263855 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21289542317390442 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9077475666999817 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9881629943847656 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2299463152885437 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2021147906780243 trainer/global_step‚9199 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23022691905498505 jet/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9156157374382019 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9899949431419373 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.669242262840271 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 _runtime‚8348.264670371 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6884473562240601 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21612626314163208 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21517635881900787 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9042706489562988 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45442554354667664 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15618422627449036 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5764763951301575 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 *jet/accuracy_2_of_4‚0.7587796449661255 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2051434963941574 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43981480598449707 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6830708980560303 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_4‚0.9965986609458923 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487;}ôÛÎ957 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21786759793758392 particle/accuracy_2_of_4‚1 _step‚758 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.651515007019043 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 epoch‚22 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9894712567329407 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2877899408340454 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06318183988332748 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6204833984375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.26622438430786133 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6185691356658936 +jet/accuracy_4_of_4‚0.01670061983168125 (particle/accuracy‚0.8493186235427856 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03976878523826599öê…dü‚(:Ù í”öÒêÛËË disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚59431.58740234375 disk.rbd0.out‚0 network.sent‚ 1.4693566e+07( disk./.usagePercent‚75.9894811608197 disk.nvme1c1n1.in‚0' disk.nvme0c0n1.out‚46933.9208984375$ memory_percent‚96.78723812103271( proc.memory.availableMB‚ 526.37890625* proc.memory.percent‚18.447065353393555 proc.cpu.threads‚17& disk.nvme1n1.out‚24803.67822265625 disk.md0.out‚ 248.6015625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 0.00390625% disk./.usageGB‚1166.6285095214844% disk.nvme0n1.out‚46933.9208984375" disk.dm-1.out‚11774.1962890625 disk.rbd0.in‚0" proc.memory.rssMB‚ 3022.3671875 cpu‚25.69623411161288 network.recv‚ 2.22564e+06 disk.nvme0n1.in‚ 0.00390625( disk.nvme1c1n1.out‚24803.67822265625 disk.nvme1n1.in‚0 disk.md0.in‚0#F,úžý‚(:“ í”öÒä‰ÌÌ gpu.0.gpu‚94.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.675' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.55833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=ÈŒÁbþ‚(:× ü”öҵܰË) disk./.usagePercent‚75.98948240318975 disk.md0.in‚0 disk.rbd0.out‚0( disk.nvme1c1n1.out‚24804.39599609375$ memory_percent‚96.78740501403809" proc.memory.rssMB‚ 3022.3671875* proc.memory.percent‚18.447065353393555 network.sent‚ 1.4752404e+07% disk./.usageGB‚1166.6285285949707 disk.md0.out‚ 249.140625# disk.dm-0.out‚59432.16552734375 disk.dm-1.in‚0" disk.dm-1.out‚11774.2509765625& disk.nvme0c0n1.out‚46934.998046875$ disk.nvme0n1.out‚46934.998046875 disk.dm-0.in‚ 0.00390625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚24804.39599609375' proc.memory.availableMB‚ 526.3515625 network.recv‚ 2.229933e+06 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 cpu‚25.69585466018417 proc.cpu.threads‚17à¿Úzžÿ‚(:“ ü”öÒØ¿ñË gpu.0.gpu‚97.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.311' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.10366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ȼf€‚(:Û ‹•öÒŽï×Ë disk.dm-1.in‚0# disk.dm-1.out‚11774.34130859375' disk.nvme0c0n1.out‚46936.3251953125% disk.nvme0n1.out‚46936.3251953125( disk.nvme1c1n1.out‚24805.06396484375% disk./.usageGB‚1166.6285858154297" disk.dm-0.out‚59433.0947265625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0" proc.memory.rssMB‚ 3022.3671875* proc.memory.percent‚18.447065353393555 network.sent‚ 1.4757251e+07 network.recv‚ 2.233352e+06 disk.dm-0.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1n1.out‚24805.06396484375) disk./.usagePercent‚75.98948613029994 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625$ memory_percent‚96.78773880004883& proc.memory.availableMB‚ 526.296875 cpu‚25.696038641984245 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚ 249.59765625» !A“‚(:ˆ ‹•öÒìþ•Ì gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.014' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.338 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥^Žë2‚- _wandbruntime‚8391  _runtime‚8391s~Ùȃ trainer/global_step‚9249 $ _timestamp‚1.784515217766956e+09 _runtime‚8391.342633593 _step‚759 ' lr-AdamW/pg1‚0.00035389728390045256 ' lr-AdamW/pg2‚0.00035389728390045256ž^®Í„Ç ' lr-AdamW/pg2‚0.00035389728390045256 trainer/global_step‚9249 $ _timestamp‚1.784515217766956e+09 _runtime‚8391.342633593 _step‚759 ' lr-AdamW/pg1‚0.00035389728390045256÷^Ÿäñ2…- _wandbruntime‚8391  _runtime‚8391ÔØy(†úO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10748675167232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10809416504320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1426996962761216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74913739 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1416248646057472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10748675167232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10748333743104 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74913737 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1418690588416583 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10748333743104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166456153 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91532271 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1426997321224704 trainer/global_step‚9249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16139784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38246182 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307565056059904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307565037447680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91542414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 % _timestamp‚1.7845152177691495e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1416248629018112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚318374359168512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10732122415581 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 166446008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚318374472564224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1416248646057472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38245909 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1426996962761216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91532271 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22106128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74913737 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91542414 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1407958108400368 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10732462976855 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22106398 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚8391.344807687 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1426996962761216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91532271 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22106128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74913737 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91542414 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1407958108400368 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10732462976855 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22106398 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚8391.344807687 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16139784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38246182 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307565056059904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307565037447680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91542414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 % _timestamp‚1.7845152177691495e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799øÜk¨2ˆ- _wandbruntime‚8391  _runtime‚8391qBž‰‚N NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91552084 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166473616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166463457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22108424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚318408660651008 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1427146347362304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1418838769871673 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1416396920559616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚ 8391.96976786 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38249939 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88484864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16141515 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10733571365219 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74921532 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166473616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10810442933248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10733230803833 trainer/global_step‚9249 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1416396903520256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10749443842048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74921532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107097088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1416396903520256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74921530 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10810531418112 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1418839127472419 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38250216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚761 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91552084 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ƒ–¦ü 'Š‡N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22108698 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307598236329984 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10810531418112 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1418839127472419 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38250216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚761 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91552084 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91552084 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166473616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 166463457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22108424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚318408660651008 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1427146347362304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1418838769871673 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1416396920559616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚ 8391.96976786 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38249939 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88484864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16141515 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10733571365219 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74921532 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166473616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10810442933248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10733230803833 trainer/global_step‚9249 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1416396903520256 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10749443842048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74921532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107097088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1416396903520256 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74921530 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 QDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚9299 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1423662354125312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784515262152229e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10803902844928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167330455 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22220945 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38447426 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75303389 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1423662371164672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1426117605003591 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1423662371164672 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1434466256970240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1423662354125312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92027066 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚309224069563904 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10865201117696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1434466615433728 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1434466256970240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16226484 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10803902844928 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309224050951680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92016923 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75303389 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75303387«:†BÉ ™þO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92016923 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75303389 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75303387 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚320089252069376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38447709 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22221225 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1415329658813328 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10787946190263 _runtime‚ 8435.73108472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92016923 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10865304289792 _step‚7649¬ÿ¢; F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167320310 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10787605628989 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1415329641773968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92027066 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10804244269056 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚9299 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1423662354125312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784515262152229e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10803902844928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167330455 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22220945 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38447426 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75303389 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1423662371164672 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1426117605003591 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1423662371164672 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1434466256970240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1423662354125312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92027066 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚309224069563904 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10865201117696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1434466615433728 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1434466256970240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16226484 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10803902844928 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309224050951680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496ü²¹ö2š- _wandbruntime‚8436  _runtime‚8436…ÆÖ '›…N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10789054578627 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75311182 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92036736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1415477072441440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _runtime‚8436.350126934 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10866424583680 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1423810645666816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1434616000041984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10805354375168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167347918 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚320123674417664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309257231221760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1426265786458681 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167347918 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10866308835840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75311180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38451663 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22223445 _step‚765 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75311180 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10805012943872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38451366 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1415477089480800 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1423810628627456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1423810645666816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1434616000041984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10805012943872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚320123540057600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75311182 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22223151 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 trainer/global_step‚9299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92036736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10805354375168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92026579 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10788714017241 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224“·>'œŠN B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1423810628627456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1426266144059427 % _timestamp‚1.7845152627740953e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134360064 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1434615641571328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167337759 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38451366 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1415477089480800 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1423810628627456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1423810645666816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1434616000041984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10805012943872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚320123540057600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75311182 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22223151 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 trainer/global_step‚9299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92036736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10805354375168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92026579 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10788714017241 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10789054578627 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75311182 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92036736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1415477072441440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _runtime‚8436.350126934 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10866424583680 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1423810645666816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1434616000041984 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10805354375168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167347918 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚320123674417664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309257231221760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1426265786458681 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167347918 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10866308835840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75311180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38451663 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22223445 _step‚765 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75311180 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10805012943872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FǧÁ _runtime‚8479.055729381 _step‚767 & lr-AdamW/pg1‚0.0003490239199108993 & lr-AdamW/pg2‚0.0003490239199108993 trainer/global_step‚9349 % _timestamp‚1.7845153054800162e+09›P“cÌ¨Æ % _timestamp‚1.7845153054800162e+09 _runtime‚8479.055729381 _step‚767 & lr-AdamW/pg1‚0.0003490239199108993 & lr-AdamW/pg2‚0.0003490239199108993 trainer/global_step‚9349ÿ­$T2©- _wandbruntime‚8479  _runtime‚8479c`8n(ªûO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1422701175147568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92511718 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1441935551179264 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10859813370880 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92501575 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22334920 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1422701192186928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1441935909642752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16313184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚321804150216192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75693037 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1431076096271872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚310883064455680 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1441935551179264 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10859813370880 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10859471946752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10859471946752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1433544263989965 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75693039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚9349 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10843429403671 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92511718 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1433544621590599 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1431076079232512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚321804274097664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10921085760512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7845153054822109e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1441935909642752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 _runtime‚8479.061077263 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168204757 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75693037 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168194612 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38648395 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38648101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22335211 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184Ÿã¥(«€P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1431076096271872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚310883064455680 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1441935551179264 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10859813370880 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10859471946752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10859471946752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1433544263989965 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75693039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚9349 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10843429403671 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92511718 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1433544621590599 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1431076079232512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚321804274097664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10921085760512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7845153054822109e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1441935909642752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 _runtime‚8479.061077263 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168204757 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75693037 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168194612 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38648395 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38648101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22335211 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1422701175147568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92511718 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1441935551179264 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10859813370880 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92501575 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22334920 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1422701192186928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1441935909642752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16313184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚321804150216192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75693037€“4‹2¬- _wandbruntime‚8479  _runtime‚8479\ý—± '­ƒN K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38652074 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16314918 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 8479.68100652 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75700830 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75700832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1433693160646435 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10860923476992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 _step‚769 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10860582045696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22337453 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92521388 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10922208002048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1442085294251008 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1431224353734656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168212061 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1433692803045689 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10844537792035 $ _timestamp‚1.784515306105269e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1442084935780352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10860923476992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10860582045696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168222220 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92521388 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10922309069824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168212061 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚321838572407808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92511231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚321838452727808 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92511231 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168222220 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22337159 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10844197230649 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚9349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75700832 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1431224370774016 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10860923476992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 _step‚769 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚挷èÁ!10860582045696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22337453 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92521388 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10922208002048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1442085294251008 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1431224353734656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168212061 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1433692803045689 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10844537792035 $ _timestamp‚1.784515306105269e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1442084935780352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10860923476992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10860582045696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168222220 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92521388 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10922309069824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168212061 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚321838572407808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92511231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚321838452727808 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92511231 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168222220 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22337159 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10844197230649 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚9349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75700832 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1431224370774016 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38652074 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16314918 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 8479.68100652 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75700830 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360çCÁ2¯- _wandbruntime‚8479  _runtime‚8479œrÅf °ƒ _runtime‚8479.681633962 2loss/FBt1/assignment_loss‚0.062280071404052055 1loss/FBt2/assignment_loss‚0.01875432750348865 _step‚770 2loss/SRqqt2/assignment_loss‚0.5212546921254575 /loss/FRt2/detection_loss‚1.3900222484274407 %loss/total_loss‚7.839534274216776 0loss/FRt1/assignment_loss‚1.6464946260394704 3loss/SRqqt1/assignment_loss‚0.49584394934467146 1loss/SRqqt1/detection_loss‚0.6904740476279558 /loss/FBt2/detection_loss‚0.5677871366731837 trainer/global_step‚9349 $ _timestamp‚1.784515306106318e+09 0loss/FRt2/assignment_loss‚0.8127576386197677 /loss/FRt1/detection_loss‚0.4831284721714336 1loss/SRqqt2/detection_loss‚0.6346379787804738 /loss/FBt1/detection_loss‚0.5160990854993803 epoch‚23!ÁœÅ±ˆ 0loss/FRt2/assignment_loss‚0.8127576386197677 /loss/FRt1/detection_loss‚0.4831284721714336 1loss/SRqqt2/detection_loss‚0.6346379787804738 /loss/FBt1/detection_loss‚0.5160990854993803 epoch‚23 _runtime‚8479.681633962 2loss/FBt1/assignment_loss‚0.062280071404052055 1loss/FBt2/assignment_loss‚0.01875432750348865 _step‚770 2loss/SRqqt2/assignment_loss‚0.5212546921254575 /loss/FRt2/detection_loss‚1.3900222484274407 %loss/total_loss‚7.839534274216776 0loss/FRt1/assignment_loss‚1.6464946260394704 3loss/SRqqt1/assignment_loss‚0.49584394934467146 1loss/SRqqt1/detection_loss‚0.6904740476279558 /loss/FBt2/detection_loss‚0.5677871366731837 trainer/global_step‚9349 $ _timestamp‚1.784515306106318e+09‚†Ê- a²‚(:Ö ô•öÒì¯×Ë disk.dm-0.in‚ 0.00390625# disk.dm-0.out‚59440.83935546875% proc.memory.availableMB‚ 524.53125 network.sent‚ 1.4916663e+07 disk.rbd0.out‚0% disk.nvme0n1.out‚46945.7919921875 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.dm-1.in‚0# disk.dm-1.out‚11774.98291015625* proc.memory.percent‚18.447065353393555 network.recv‚ 2.257425e+06) disk./.usagePercent‚75.98948165776773 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0& disk.nvme1c1n1.out‚24809.654296875$ memory_percent‚96.79851531982422" proc.memory.rssMB‚ 3022.3671875$ disk./.usageGB‚1166.628517150879 disk.md0.out‚ 252.21484375 cpu‚25.695058902011965 proc.cpu.threads‚17$ disk.nvme1n1.out‚24809.654296875' disk.nvme0c0n1.out‚46945.7919921875 disk.nvme0n1.in‚ 0.00390625w:«D›³‚(: ô•öÒ£ú—Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.96' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.98666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X ûa´‚(:Ö ƒ–öÒ®«–Ë disk.dm-0.in‚ 0.00390625% disk./.usageGB‚1166.6289596557617# disk.dm-1.out‚11775.07275390625 disk.nvme0c0n1.in‚ 0.00390625% disk.nvme0c0n1.out‚46946.66796875' proc.memory.availableMB‚ 524.8828125* proc.memory.percent‚18.447065353393555 network.sent‚ 1.4966275e+07# disk.nvme0n1.out‚46946.66796875' disk.nvme1c1n1.out‚24810.2763671875" proc.memory.rssMB‚ 3022.3671875 cpu‚25.694601412756146 proc.cpu.threads‚17 network.recv‚ 2.261603e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 252.64453125# disk.dm-0.out‚59441.32373046875# memory_percent‚96.7963695526123) disk./.usagePercent‚75.98951048075308 disk.dm-1.in‚0 disk.nvme0n1.in‚ 0.00390625 disk.nvme1c1n1.in‚0% disk.nvme1n1.out‚24810.2763671875%…@žµ‚(:“ ƒ–öÒ§·ìË gpu.0.gpu‚94.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚280.772' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.59066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ǻ5f¶‚(:Û ’–öÒ“ø¨Ë network.sent‚ 1.4972166e+07 network.recv‚ 2.264706e+06) disk./.usagePercent‚75.98951594718135% disk./.usageGB‚1166.6290435791016% disk.nvme1n1.out‚24811.1845703125 disk.md0.out‚ 253.1328125 disk.nvme0c0n1.in‚ 0.00390625 disk.nvme1n1.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚46949.36669921875' disk.nvme1c1n1.out‚24811.1845703125( proc.memory.availableMB‚ 524.74609375* proc.memory.percent‚18.447065353393555 cpu‚25.69411210236607 disk.md0.in‚0 disk.dm-0.in‚ 0.00390625& disk.nvme0n1.out‚46949.36669921875 disk.nvme1c1n1.in‚0" proc.memory.rssMB‚ 3022.3671875" disk.dm-0.out‚59443.8173828125# disk.dm-1.out‚11775.14306640625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 0.00390625$ memory_percent‚96.79720401763916 proc.cpu.threads‚17 disk.rbd0.out‚0²Ö7äž·‚(:“ ’–öÒ•ƒ„Ì gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚268.504' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.50133333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aÝ2¸- _wandbruntime‚8522  _runtime‚8522CW¶lǹÁ & lr-AdamW/pg1‚0.0003465831345179729 & lr-AdamW/pg2‚0.0003465831345179729 trainer/global_step‚9399 % _timestamp‚1.7845153493422487e+09 _runtime‚8522.917815296 _step‚771 '&LÌºÆ _runtime‚8522.917815296 _step‚771 & lr-AdamW/pg1‚0.0003465831345179729 & lr-AdamW/pg2‚0.0003465831345179729 trainer/global_step‚9399 % _timestamp‚1.7845153493422487e+09ƒrDJ2»- _wandbruntime‚8522  _runtime‚8522“»:ø(¼ûO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1438489821379072 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1438489804339712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚8522.922093765 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10915041048576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10915382472704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16399884 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1438489804339712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚312542096571904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1440971280576973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1449405203851776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169079059 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1430072725560528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10898912617079 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169068914 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76082687 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚323519041805312 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10898572055805 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76082689 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22450116 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10915382472704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92986227 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 trainer/global_step‚9399 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1449404845388288 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10976963845632 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10915041048576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16399881 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10977071212032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1449404845388288 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1440971638177607 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92986227 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38850000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38849711 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚772 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92996370 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1430072708521168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76082687 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323519167783936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1449405203851776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92996370 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1438489821379072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232ã²o½€P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10915041048576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16399881 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10977071212032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1449404845388288 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1440971638177607 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92986227 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38850000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38849711 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚772 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92996370 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1430072708521168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76082687 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323519167783936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1449405203851776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92996370 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1438489821379072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1438489821379072 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1438489804339712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚8522.922093765 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10915041048576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10915382472704 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16399884 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1438489804339712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚312542096571904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1440971280576973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1449405203851776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169079059 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMoni •Vkw tor.on_train_batch_start/requested_bytes.large_pool.allocated‚1430072725560528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10898912617079 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169068914 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76082687 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚323519041805312 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10898572055805 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76082689 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22450116 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10915382472704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92986227 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 trainer/global_step‚9399 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1449404845388288 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10976963845632 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128„>„‡2¾- _wandbruntime‚8523  _runtime‚85237~H '¿…N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22452114 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1441119819632697 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1441120177233443 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169086363 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1449554229989376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76090480 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1438638078841856 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10916492578816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38853729 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169096522 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93006040 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169096522 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76090482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚312575276841984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚8523.541558333 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _step‚773 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1449554588460032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚323553336310272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10916151147520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76090480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10978183342592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚323553460184576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 % _timestamp‚1.7845153499658105e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38854020 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10978078080512 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1430220139188640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10899680444057 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10916492578816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92995883 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1430220156228000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1449554588460032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16401618 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22452402 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1438638095881216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92995883 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76090482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16401615.ôu‹'ÀŠN \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚323553460184576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 % _timestamp‚1.7845153499658105e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38854020 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10978078080512 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1430220139188640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10899680444057 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10916492578816 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92995883 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1430220156228000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1449554588460032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16401618 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22452402 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1438638095881216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92995883 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76090482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16401615 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22452114 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1441119819632697 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1441120177233443 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169086363 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1449554229989376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76090480 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1438638078841856 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10916492578816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38853729 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169096522 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93006040 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169096522 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76090482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚312575276841984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚8523.541558333 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _step‚773 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1449554588460032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚323553336310272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10916151147520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76090480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10978183342592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0…*1ßA2Á- _wandbruntime‚8523  _runtime‚8523íß8ÿ Â… 0loss/FRt1/assignment_loss‚1.7128144885558056 /loss/FRt2/detection_loss‚1.4979752415862795 epoch‚23 % _timestamp‚1.7845153499668634e+09 /loss/FBt1/detection_loss‚0.5367868882119969 0loss/FBt2/detection_loss‚0.27500668964348735 _runtime‚8523.542149874 0loss/FRt2/assignment_loss‚0.7429247443492596 2loss/SRqqt1/assignment_loss‚0.4708666599115826 2loss/SRqqt2/assignment_loss‚0.5308284763277684 4loss/FBt2/assignment_loss‚0.00012730040905157837 2loss/FBt1/assignment_loss‚0.007227139561173282 /loss/FRt1/detection_loss‚0.4644514383903498 0loss/SRqqt1/detection_loss‚0.735105505803966 0loss/SRqqt2/detection_loss‚0.611405164108334 %loss/total_loss‚7.585519736859054 trainer/global_step‚9399 _step‚774ÄçÀÊ /loss/FBt1/detection_loss‚0.5367868882119969 0loss/FBt2/detection_loss‚0.27500668964348735 _runtime‚8523.542149874 0loss/FRt2/assignment_loss‚0.7429247443492596 2loss/SRqqt1/assignment_loss‚0.4708666599115826 2loss/SRqqt2/assignment_loss‚0.5308284763277684 4loss/FBt2/assignment_loss‚0.00012730040905157837 2loss/FBt1/assignment_loss‚0.007227139561173282 /loss/FRt1/detection_loss‚0.4644514383903498 0loss/SRqqt1/detection_loss‚0.735105505803966 0loss/SRqqt2/detection_loss‚0.611405164108334 %loss/total_loss‚7.585519736859054 trainer/global_step‚9399 _step‚774 0loss/FRt1/assignment_loss‚1.7128144885558056 /loss/FRt2/detection_loss‚1.4979752415862795 epoch‚23 % _timestamp‚1.7845153499668634e+09†z¢…à{Ä‚(:ð ¡–öÒ¡íÇË network.recv‚ 2.268213e+06& disk.nvme1n1.out‚24874.59716796875 disk.md0.out‚ 253.62109375" disk.dm-0.out‚59687.3798828125 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚47130.72900390625& disk.nvme0n1.out‚47130.72900390625 network.sent‚ 1.4989824e+07$ disk./.usageGB‚1166.867073059082 disk.dm-0.in‚ 166.46484375 disk.dm-1.in‚0! disk.dm-1.out‚11775.310546875 disk.nvme0n1.in‚ 62.63671875( proc.memory.availableMB‚ 524.69140625" proc.memory.rssMB‚ 3022.3671875) disk./.usagePercent‚76.00502022856746* proc.memory.percent‚18.447065353393555 cpu‚25.69368236368991 proc.cpu.threads‚17 disk.nvme1n1.in‚ 103.828125 disk.md0.in‚0! disk.nvme0c0n1.in‚ 62.63671875 disk.nvme1c1n1.in‚ 103.828125( disk.nvme1c1n1.out‚24874.59716796875# memory_percent‚96.7975378036499ÔéžÅ‚(:“ ¡–öÒ§ùˆÌ gpu.0.gpu‚85.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.247' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.41566666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)Õ©~Æ‚(:ó °–öÒõ²¡Ë& disk.nvme0n1.out‚47135.00830078125! disk.nvme1c1n1.in‚ 168.5703125! disk.dm-1.out‚11775.357421875 disk.rbd0.out‚0$ memory_percent‚96.79903984069824 network.recv‚ 2.272369e+06" disk.nvme0c0n1.in‚ 129.14453125 disk.nvme1n1.in‚ 168.5703125& disk.nvme1n1.out‚24875.13720703125 disk.md0.in‚0 disk.dm-0.in‚ 297.71484375! disk.dm-0.out‚59691.673828125 proc.cpu.threads‚17( disk.nvme1c1n1.out‚24875.13720703125 disk.rbd0.in‚0* proc.memory.percent‚18.447065353393555 network.sent‚ 1.5038314e+07% disk./.usageGB‚1164.1509895324707 disk.md0.out‚ 253.8359375 disk.dm-1.in‚0' proc.memory.availableMB‚ 524.4453125" proc.memory.rssMB‚ 3022.3671875 cpu‚25.693228366823103( disk./.usagePercent‚75.8281054898206( disk.nvme0c0n1.out‚47135.00830078125 disk.nvme0n1.in‚ 129.14453125KTO5žÇ‚(:“ °–öÒèÊüË gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚268.634' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.54466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ÅwxÈ‚(:í ¿–öÒ¨«ŽË* proc.memory.percent‚18.447065353393555 network.recv‚ 2.276013e+06$ disk./.usageGB‚1164.150478363037! disk.nvme0c0n1.in‚ 129.1953125 disk.nvme0n1.in‚ 129.1953125& disk.nvme0n1.out‚47136.47021484375$ memory_percent‚96.79751396179199 proc.cpu.threads‚17 network.sent‚ 1.5044306e+07) disk./.usagePercent‚75.82807219430302 disk.nvme1n1.in‚ 169.84765625 disk.dm-0.in‚ 299.04296875# disk.dm-1.out‚11775.50439453125 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚47136.47021484375" disk.nvme1n1.out‚ 24875.9765625 disk.md0.in‚0" disk.nvme1c1n1.in‚ 169.84765625$ disk.nvme1c1n1.out‚ 24875.9765625' proc.memory.availableMB‚ 524.6953125 cpu‚25.69277501536791 disk.md0.out‚ 254.40234375! disk.dm-0.out‚59692.615234375 disk.dm-1.in‚0" proc.memory.rssMB‚ 3022.3671875Ò¯¼{“É‚(:ˆ ¿–öÒ•µæË gpu.0.gpu‚57.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚198.669' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚66.223 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿Å´•2Ê- _wandbruntime‚8567  _runtime‚8567ÿ¨ïÒÇËÁ $ _timestamp‚1.784515393940702e+09 _runtime‚ 8567.51707273 _step‚775 ' lr-AdamW/pg1‚0.00034413982249160996 ' lr-AdamW/pg2‚0.00034413982249160996 trainer/global_step‚9449ZR@‹ÌÌÆ ' lr-AdamW/pg1‚0.00034413982249160996 ' lr-AdamW/pg2‚0.00034413982249160996 trainer/global_step‚9449 $ _timestamp‚1.784515393940702e+09 _runtime‚ 8567.51707273 _step‚775‡DÞ2Í- _wandbruntime‚8567  _runtime‚8567 ”Æ:ÎúO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚325234018749952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10970610150400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93470879 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39050027 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76472339 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76472337 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1448398297163981 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10970951574528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1448398654764615 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1437444241894768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1456874139597312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10970951574528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚297 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10954055269213 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845153939414208e+09 _step‚776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76472339 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240è©(ÏÿO VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1448398297163981 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10970951574528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1448398654764615 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1437444241894768 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1456874139597312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10970951574528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚297 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10954055269213 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845153939414208e+09 _step‚776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76472339 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚325234018749952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10970610150400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93470879 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39050027 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76472339 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76472337ˆ A …2Ð- _wandbruntime‚8568  _runtime‚8568$$ 'уN UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1446051803949056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16488318 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76480130 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1446051803949056 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10971720249344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1457023882669056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169960665 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11033931271168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1457023882669056 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169960665 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93490692 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39054016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314234290345984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10971720249344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10955163657465 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22565965 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16488315 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93480535 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1457023524198400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10955504218851 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22565701 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314234271733760 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1437591689601600 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1446051820988416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76480132 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93490692 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39054283 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚325268312199168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚325268203004928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1448546836219705 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _step‚777 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚9449 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10972061680640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169970824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10972061680640 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1457023524198400 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10955504218851 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22565701 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314234271733760 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1437591689601600 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1446051820988416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76480132 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93490692 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39054283 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚325268312199168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚325268203004928 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1448546836219705 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 _step‚777 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚9449 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10972061680640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169970824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10972061680640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76480130 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1446051803949056 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10971720249344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1457023882669056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169960665 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11033931271168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1457023882669056 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169960665 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93490692 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39054016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314234290345984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10971720249344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528‰6”Ï2Ó- _wandbruntime‚8568  _runtime‚8568ãÙêQ Ôƒ 2loss/SRqqt1/assignment_loss‚0.4590199343464916 /loss/FBt1/detection_loss‚0.5695056350052576 %loss/total_loss‚7.741485019552529 _step‚778 0loss/FRt2/assignment_loss‚0.7154610452455875 3loss/FBt2/assignment_loss‚0.0007528874859125188 epoch‚23 /loss/FRt1/assignment_loss‚1.722780665976251 2loss/SRqqt2/assignment_loss‚0.5238718039245791 2loss/FBt1/assignment_loss‚0.024874888402256354 .loss/FRt1/detection_loss‚0.503232293080614 1loss/SRqqt1/detection_loss‚0.7709375943260361 /loss/FBt2/detection_loss‚0.3834272559478824 _runtime‚8568.141318903 /loss/FRt2/detection_loss‚1.4011748306598368 1loss/SRqqt2/detection_loss‚0.6664461851518249 trainer/global_step‚9449 % _timestamp‚1.7845153945660353e+09•2™Õˆ /loss/FBt1/detection_loss‚0.5695056350052576 %loss/total_loss‚7.741485019552529 _step‚778 0loss/FRt2/assignment_loss‚0.7154610452455875 3loss/FBt2/assignment_loss‚0.0007528874859125188 epoch‚23 /loss/FRt1/assignment_loss‚1.722780665976251 2loss/SRqqt2/assignment_loss‚0.5238718039245791 2loss/FBt1/assignment_loss‚0.024874888402256354 .loss/FRt1/detection_loss‚0.503232293080614 1loss/SRqqt1/detection_loss‚0.7709375943260361 /loss/FBt2/detection_loss‚0.3834272559478824 _runtime‚8568.141318903 /loss/FRt2/detection_loss‚1.4011748306598368 1loss/SRqqt2/detection_loss‚0.6664461851518249 trainer/global_step‚9449 % _timestamp‚1.7845153945660353e+09 2loss/SRqqt1/assignment_loss‚0.4590199343464916Šþ¹¤ËyÖ‚(:î ΖöÒš»ÊË network.sent‚ 1.506199e+07 disk.dm-0.in‚ 325.71484375' disk.nvme0c0n1.out‚47148.7353515625 cpu‚25.692290261952348) disk./.usagePercent‚75.81259102099999# disk.dm-1.out‚11775.69189453125 disk.rbd0.in‚0' disk.nvme1c1n1.out‚24878.3056640625 disk.nvme1n1.in‚ 190.734375 disk.md0.out‚ 255.0546875$ proc.memory.availableMB‚526.1875" proc.memory.rssMB‚ 3022.3671875% disk.nvme0n1.out‚47148.7353515625 network.recv‚ 2.279324e+06# disk.dm-0.out‚59705.64501953125 disk.dm-1.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 134.98046875 disk.nvme1c1n1.in‚ 190.734375 disk.md0.in‚0% disk./.usageGB‚1163.9128036499023% disk.nvme1n1.out‚24878.3056640625$ memory_percent‚96.78840637207031* proc.memory.percent‚18.447065353393555 proc.cpu.threads‚17 disk.nvme0n1.in‚ 134.98046875Óe«ž×‚(:“ ΖöÒˆÛ‘Ì gpu.0.gpu‚46.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.327' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.44233333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘1©‘pØ‚(:å Ý–öÒ”ÁºË( disk.nvme1c1n1.out‚24878.92529296875 disk.nvme1n1.in‚ 190.734375& disk.nvme1n1.out‚24878.92529296875 disk.md0.out‚ 255.4765625 disk.dm-1.out‚ 11775.78125" proc.memory.rssMB‚ 3022.3671875* proc.memory.percent‚18.447065353393555 proc.cpu.threads‚17 network.sent‚ 1.5110565e+07 network.recv‚ 2.284071e+06 disk.nvme0n1.in‚ 134.98046875! disk.nvme0n1.out‚ 47149.703125 disk.md0.in‚0# disk.dm-0.out‚59706.23095703125 disk.rbd0.in‚0 disk.rbd0.out‚0( disk./.usagePercent‚75.7971210290275 disk.nvme1c1n1.in‚ 190.734375" disk.nvme0c0n1.in‚ 134.98046875# disk.nvme0c0n1.out‚ 47149.703125# memory_percent‚96.7883825302124( proc.memory.availableMB‚ 526.19140625% disk./.usageGB‚1163.6753005981445 disk.dm-0.in‚ 325.71484375 disk.dm-1.in‚0 cpu‚25.6918109283952522¼:èÙ‚(:’ Ý–öÒìÉ”Ì gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚275.93' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.97666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üL{Ú‚(:ð ì–öÒ„íËË network.recv‚ 2.287037e+06) disk./.usagePercent‚75.78164010419847 disk.nvme1n1.in‚ 190.734375 disk.md0.in‚0( disk.nvme1c1n1.out‚24882.19677734375$ memory_percent‚96.78843021392822" proc.memory.rssMB‚ 3022.3671875 network.sent‚ 1.5117457e+07 disk.dm-1.in‚0 disk.nvme0n1.in‚ 134.98046875 cpu‚25.691331636764613 proc.cpu.threads‚17 disk.rbd0.out‚0$ disk./.usageGB‚1163.437629699707& disk.nvme1n1.out‚24882.19677734375 disk.dm-0.in‚ 325.71484375 disk.dm-1.out‚11775.83203125' disk.nvme0c0n1.out‚47165.4580078125% disk.nvme0n1.out‚47165.4580078125( proc.memory.availableMB‚ 526.18359375 disk.md0.out‚ 255.984375# disk.dm-0.out‚59724.13232421875" disk.nvme0c0n1.in‚ 134.98046875 disk.nvme1c1n1.in‚ 190.734375* proc.memory.percent‚18.447065353393555 disk.rbd0.in‚0ùýfJŸÛ‚(:” ì–öÒÐäŽÌ gpu.0.gpu‚57.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.104' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.367999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÇú2Ü- _wandbruntime‚8610  _runtime‚8610ŒZ™ÉÝà _runtime‚8610.856015977 _step‚779 ' lr-AdamW/pg1‚0.00034169414076117596 ' lr-AdamW/pg2‚0.00034169414076117596 trainer/global_step‚9499 % _timestamp‚1.7845154372804236e+09kÏ›ÎÞÈ trainer/global_step‚9499 % _timestamp‚1.7845154372804236e+09 _runtime‚8610.856015977 _step‚779 ' lr-AdamW/pg1‚0.00034169414076117596 ' lr-AdamW/pg2‚0.00034169414076117596‹è°°2ß- _wandbruntime‚8610  _runtime‚8610¥ ¼d(àúO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326948747118080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 $ _timestamp‚1.784515437281454e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11026520676352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76861987 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1444815775268368 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11026179252224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16573284 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1464343792269824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93955531 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11088751613952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚315860104967680 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1455825671351623 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1464343433806336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1453317254554112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76861989 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326948875193856 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11026520676352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚315860123579904 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1464343433806336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11009538482621 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11026179252224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚8610.857100232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11009879043895 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16573281 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11088642150400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76861987 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1455825313750989 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚9499 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1464343792269824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93965674¦L;(áÿO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170827663 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39250429 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11026520676352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76861987 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1444815775268368 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11026179252224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16573284 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1464343792269824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93955531 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11088751613952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚315860104967680 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1455825671351623 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1464343433806336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1453317254554112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76861989 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚326948875193856 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11026520676352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚315860123579904 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1464343433806336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11009538482621 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11026179252224 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚8610.857100232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11009879043895 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16573281 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11088642150400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76861987 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1455825313750989 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚9499 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1464343792269824 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93965674 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326948747118080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 $ _timestamp‚1.784515437281454e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11010987432259 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11027289351168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170845126 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39254161 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_ûCÖ^yend/inactive_split_bytes.all.peak‚ 717119488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93975344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93965187 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11089868085760 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1444963205935840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8611.478752456 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93965187 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39254437 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16575015 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚9499 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76869782 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76869780 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚315893285237760 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1444963222975200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22679419 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326983045964288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1455974210407459 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170845126 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1453465546095616 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1464492818407424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 % _timestamp‚1.7845154379030182e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170834967 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1455973852806713 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76869780 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 % _timestamp‚1.7845154379030182e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170834967 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1455973852806713 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76869780 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11010987432259 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11027289351168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170845126 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39254161 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93975344 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93965187 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11089868085760 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1444963205935840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8611.478752456 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93965187 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39254437 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16575015 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0‡ìï´2å- _wandbruntime‚8611  _runtime‚8611Õë²Sæ‚ epoch‚23 /loss/FRt2/assignment_loss‚0.863657794383742 3loss/FBt1/assignment_loss‚0.0032806109894963397 .loss/FRt1/detection_loss‚0.490856160989625 /loss/FRt2/detection_loss‚1.4744191196656498 0loss/SRqqt1/detection_loss‚0.810672121505765 0loss/SRqqt2/detection_loss‚0.651939794119949 0loss/FBt1/detection_loss‚0.34083867835954973 2loss/SRqqt1/assignment_loss‚0.5382560738821907 2loss/SRqqt2/assignment_loss‚0.5743062312641111 trainer/global_step‚9499 % _timestamp‚1.7845154379040494e+09 _step‚782 $loss/total_loss‚7.84251531521986 0loss/FBt2/detection_loss‚0.47084706648609337 _runtime‚8611.479336689 0loss/FRt1/assignment_loss‚1.6039492341169628 2loss/FBt2/assignment_loss‚0.019492429456725625¼Ï×T ç‡ 0loss/FRt1/assignment_loss‚1.6039492341169628 2loss/FBt2/assignment_loss‚0.019492429456725625 epoch‚23 /loss/FRt2/assignment_loss‚0.863657794383742 3loss/FBt1/assignment_loss‚0.0032806109894963397 .loss/FRt1/detection_loss‚0.490856160989625 /loss/FRt2/detection_loss‚1.4744191196656498 0loss/SRqqt1/detection_loss‚0.810672121505765 0loss/SRqqt2/detection_loss‚0.651939794119949 0loss/FBt1/detection_loss‚0.34083867835954973 2loss/SRqqt1/assignment_loss‚0.5382560738821907 2loss/SRqqt2/assignment_loss‚0.5743062312641111 trainer/global_step‚9499 % _timestamp‚1.7845154379040494e+09 _step‚782 $loss/total_loss‚7.84251531521986 0loss/FBt2/detection_loss‚0.47084706648609337 _runtime‚8611.479336689Ž)M!‚è‚(:÷ û–öÒ‡ÅòË( proc.memory.availableMB‚ 526.15234375% disk./.usageGB‚1163.4375343322754 disk.nvme0n1.in‚ 134.98046875& disk.nvme0n1.out‚47166.34130859375( disk.nvme1c1n1.out‚24882.83544921875 disk.nvme1n1.in‚ 190.734375 proc.cpu.threads‚17) disk./.usagePercent‚75.78163389234818 disk.nvme1c1n1.in‚ 190.734375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚47166.34130859375" proc.memory.rssMB‚ 3022.3671875* proc.memory.percent‚18.447065353393555 cpu‚25.690939773138908 network.sent‚ 1.5133228e+07& disk.nvme1n1.out‚24882.83544921875 disk.md0.in‚0 disk.dm-0.in‚ 325.71484375# disk.dm-0.out‚59724.72607421875 disk.dm-1.in‚0# disk.dm-1.out‚11776.05615234375 disk.rbd0.out‚0 network.recv‚ 2.290664e+06 disk.md0.out‚ 256.30859375" disk.nvme0c0n1.in‚ 134.98046875# memory_percent‚96.7886209487915ª_‡žé‚(:“ û–öÒæ±­Ì gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚225.298' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚75.09933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Îçàzê‚(:ï Š—öÒ÷“¸Ë* proc.memory.percent‚18.447065353393555 cpu‚25.690467329159567 disk.md0.in‚0 disk.md0.out‚ 256.71875 network.sent‚ 1.5184952e+07% disk./.usageGB‚1165.8463706970215' disk.nvme0c0n1.out‚48554.9189453125 disk.nvme0n1.in‚ 134.98046875 disk.nvme1n1.in‚ 190.734375% disk.nvme1n1.out‚24917.8251953125% disk.nvme0n1.out‚48554.9189453125 disk.nvme1c1n1.in‚ 190.734375' disk.nvme1c1n1.out‚24917.8251953125 disk.dm-0.in‚ 325.71484375# disk.dm-0.out‚61144.87255859375# disk.dm-1.out‚11776.09912109375 disk.rbd0.in‚0$ memory_percent‚96.79040908813477) disk./.usagePercent‚75.93853578876545" disk.nvme0c0n1.in‚ 134.98046875" proc.memory.rssMB‚ 3022.3671875 proc.cpu.threads‚17 network.recv‚ 2.295097e+06 disk.dm-1.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 525.859375Ó žë‚(:“ Š—öÒÉíöË gpu.0.gpu‚72.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚268.496' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.49866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥K%·2ì- _wandbruntime‚8654  _runtime‚8654süDÖÉíà ' lr-AdamW/pg1‚0.00033924624640823856 ' lr-AdamW/pg2‚0.00033924624640823856 trainer/global_step‚9549 % _timestamp‚1.7845154807550623e+09 _runtime‚8654.332060705 _step‚783Þr;ÎîÈ % _timestamp‚1.7845154807550623e+09 _runtime‚8654.332060705 _step‚783 ' lr-AdamW/pg1‚0.00033924624640823856 ' lr-AdamW/pg2‚0.00033924624640823856 trainer/global_step‚9549Ñ<¼ý2ï- _wandbruntime‚8654  _runtime‚8654“E‹Ö(ðûO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171691820 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11081748354048 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1460730979661312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39451733 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1463252687938631 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚328663703452672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1460730996700672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1471812728015360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22791461 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11082089778176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1463252330337997 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317519118471680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11081748354048 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77251639 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11144465293824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39451442 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94450326 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11082089778176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16659984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16659981 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94450326 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1471813086478848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚8654.332123516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77251637 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1471812728015360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1460730979661312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22791749 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1452187308641968 _step‚784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94440183 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77251639 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171701965 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1452187325681328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317519137083904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94440183 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11082089778176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1463252330337997 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317519118471680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11081748354048 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77251639 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11144465293824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39451442 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94450326 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11082089778176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16659984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16659981 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94450326 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1471813086478848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚8654.332123516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77251637 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1471812728015360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1460730979661312 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22791749 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1452187308641968 _step‚784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94440183 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77251639 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171701965 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1452187325681328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317519137083904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94440183 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1460730979661312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39451733 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1463252687938631 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚328663703452672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272.O2ò- _wandbruntime‚8654  _runtime‚8654¢´Âù 'ó…N RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1460879254163456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11082858452992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94449839 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134360064 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11145572694528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1471962471087104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚8654.956938747 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1460879271202816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39455711 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚317552317353984 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39455422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22793707 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77259430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11066470645667 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171719428 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1452334756348800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1471962471087104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94459996 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171709269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚328698005796352 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77259432 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1452334739309440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11066130084281 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94449839 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1463401226994467 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1471962112616448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746Îó/'ôŠN YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11066130084281 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94449839 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1463401226994467 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1471962112616448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1460879254163456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11082858452992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94449839 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134360064 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11145572694528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1471962471087104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚8654.956938747 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1460879271202816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39455711 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚317552317353984 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39455422 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22793707 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77259430 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11066470645667 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171719428 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1452334756348800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1471962471087104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94459996 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171709269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚328698005796352 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77259432 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1452334739309440‘ÕÕ%2õ- _wandbruntime‚8654  _runtime‚8654¡œðN ö† 0loss/FRt1/assignment_loss‚1.7508897807525128 0loss/FRt2/assignment_loss‚0.7765812642377627 1loss/SRqqt2/assignment_loss‚0.517939011757494 2loss/FBt1/assignment_loss‚0.024613905981533655 /loss/FBt1/detection_loss‚0.5347969675024685 %loss/total_loss‚7.752183081896234 trainer/global_step‚9549 2loss/SRqqt1/assignment_loss‚0.5056153658310768 1loss/SRqqt2/detection_loss‚0.5845240442979815 4loss/FBt2/assignment_loss‚0.00013714681095267547 0loss/FRt1/detection_loss‚0.47127128680065244 epoch‚23 _step‚786 /loss/FRt2/detection_loss‚1.4383924161730488 1loss/SRqqt1/detection_loss‚0.7547492269303895 /loss/FBt2/detection_loss‚0.3926726648203609 % _timestamp‚1.7845154813829005e+09 _runtime‚8654.958817389 ü7æ÷‹ %loss/total_loss‚7.752183081896234 trainer/global_step‚9549 2loss/SRqqt1/assignment_loss‚0.5056153658310768 1loss/SRqqt2/detection_loss‚0.5845240442979815 4loss/FBt2/assignment_loss‚0.00013714681095267547 0loss/FRt1/detection_loss‚0.47127128680065244 epoch‚23 _step‚786 /loss/FRt2/detection_loss‚1.4383924161730488 1loss/SRqqt1/detection_loss‚0.7547492269303895 /loss/FBt2/detection_loss‚0.3926726648203609 % _timestamp‚1.7845154813829005e+09 _runtime‚8654.958817389 0loss/FRt1/assignment_loss‚1.7508897807525128 0loss/FRt2/assignment_loss‚0.7765812642377627 1loss/SRqqt2/assignment_loss‚0.517939011757494 2loss/FBt1/assignment_loss‚0.024613905981533655 /loss/FBt1/detection_loss‚0.5347969675024685’sÍé¨wø‚(:ì ™—öÒñŇÌ( proc.memory.availableMB‚ 525.35546875 network.sent‚ 1.5200765e+07) disk./.usagePercent‚75.99248968415415" disk.nvme0n1.out‚ 49377.8984375 disk.nvme1n1.in‚ 190.734375$ memory_percent‚96.79348468780518* proc.memory.percent‚18.447065353393555 cpu‚25.690047499523107% disk./.usageGB‚1166.6746978759766# disk.dm-0.out‚62119.98193359375# disk.dm-1.out‚11776.25732421875" disk.nvme0c0n1.in‚ 134.98046875$ disk.nvme0c0n1.out‚ 49377.8984375 disk.nvme0n1.in‚ 134.98046875 disk.nvme1c1n1.in‚ 190.734375 proc.cpu.threads‚17 network.recv‚ 2.298548e+06 disk.dm-0.in‚ 325.71484375 disk.rbd0.in‚0 disk.md0.in‚0 disk.md0.out‚ 257.05859375" proc.memory.rssMB‚ 3022.3671875 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme1c1n1.out‚25068.361328125$ disk.nvme1n1.out‚25068.361328125nõñÀ;ù‚(:ˆ ™—öÒÞ×ÇÌ gpu.0.gpu‚99.0 gpu.0.memo™‰ ^Xry‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚277.479' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.493 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦w"Fú‚(:ô ¨—öÒ¢¡»Ë) disk./.usagePercent‚75.98947743370952% disk./.usageGB‚1166.6284523010254% disk.nvme1n1.out‚25306.2509765625 disk.rbd0.in‚0$ memory_percent‚96.79229259490967" proc.memory.rssMB‚ 3022.3671875 cpu‚25.689636663001473( disk.nvme0c0n1.out‚50045.12841796875 disk.nvme1c1n1.in‚ 190.734375 disk.nvme1n1.in‚ 190.734375 disk.md0.out‚ 257.32421875 disk.rbd0.out‚0 network.recv‚ 2.302473e+06" disk.nvme0c0n1.in‚ 134.98046875 disk.nvme0n1.in‚ 134.98046875' disk.nvme1c1n1.out‚25306.2509765625 disk.md0.in‚0! disk.dm-0.out‚63024.400390625 disk.dm-1.in‚0# disk.dm-1.out‚11776.37451171875& disk.nvme0n1.out‚50045.12841796875 disk.dm-0.in‚ 325.71484375( proc.memory.availableMB‚ 525.55078125* proc.memory.percent‚18.447065353393555 proc.cpu.threads‚17 network.sent‚ 1.5252275e+07Ô€jäû‚(:’ ¨—öÒßöüË gpu.0.gpu‚17.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.364' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.45466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õºj{ü‚(:ð ·—öÒ­…©Ë network.sent‚ 1.5257121e+07 disk.nvme1n1.in‚ 190.734375! disk.dm-0.out‚63034.064453125 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 134.98046875( disk.nvme0c0n1.out‚50046.71728515625 disk.nvme1c1n1.in‚ 190.734375& disk.nvme1n1.out‚25316.16455078125 disk.md0.in‚0 disk.md0.out‚ 258.11328125 disk.dm-1.out‚11776.56640625( disk.nvme1c1n1.out‚25316.16455078125$ memory_percent‚96.79088592529297% proc.memory.availableMB‚ 525.78125* proc.memory.percent‚18.447065353393555 network.recv‚ 2.305741e+06 disk.dm-0.in‚ 325.71484375& disk.nvme0n1.out‚50046.71728515625 cpu‚25.689195714243727) disk./.usagePercent‚75.81601176271995% disk./.usageGB‚1163.9653205871582 disk.dm-1.in‚0 disk.nvme0n1.in‚ 134.98046875" proc.memory.rssMB‚ 3022.3671875 proc.cpu.threads‚17Üý‚(:’ ·—öÒ±ÂêË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.52' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.50666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾üe«2þ- _wandbruntime‚8698  _runtime‚8698´Ì½éÈÿ _step‚787 ' lr-AdamW/pg1‚0.00033679629665647764 ' lr-AdamW/pg2‚0.00033679629665647764 trainer/global_step‚9599 $ _timestamp‚1.784515524546491e+09 _runtime‚8698.122097443:ïO.Í€ Ç ' lr-AdamW/pg1‚0.00033679629665647764 ' lr-AdamW/pg2‚0.00033679629665647764 trainer/global_step‚9599 $ _timestamp‚1.784515524546491e+09 _runtime‚8698.122097443 _step‚787“ÙìÛ]2 - _wandbruntime‚8698  _runtime‚8698„0ðö(‚ ûO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172566122 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94924835 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39653089 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22906117 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 _step‚788 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94924835 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1470679346925005 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77641289 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319178150587904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚9599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94934978 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16746681 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11137658880000 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1459558859054928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚330378417122304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11137317455872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1470679704525639 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1479282022224384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1468144704768512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16746684 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1479282022224384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39652798 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1479282380687872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1468144704768512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77641287 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1479282380687872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22906405 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94934978 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 % _timestamp‚1.7845155245476089e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11200285146624 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11137658880000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77641289 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11120845470711 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11120504909437 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1468144721807872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚330378543100928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1468144721807872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1479282022224384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1468144704768512 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16746684 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1479282022224384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39652798 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1479282380687872 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1468144704768512 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77641287 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1479282380687872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22906405 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94934978 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 % _timestamp‚1.7845155245476089e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11200285146624 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11137658880000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77641289 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11120845470711 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11120504909437 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1468144721807872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚330378543100928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1468144721807872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 _step‚788 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94924835 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1470679346925005 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344”Eup‚2„ - _wandbruntime‚8698  _runtime‚8698O]"ç!… …N J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77649080 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1479431765296128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚330412842497536 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1470828243581475 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11121953859075 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11138768986112 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1479431406825472 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚330412712331776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319211330857984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39657119 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16748418 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39656832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11201400086016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1468292996310016 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1468292979270656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22908701 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1470827885980729 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94934491 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1479431765296128 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1468292979270656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11201511639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172583571 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77649082 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1459706272683040 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172583571 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11138768986112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11138427554816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77649080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚789 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172593730 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94944648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94934491 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1468292996310016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94944648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22908417 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11138427554816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172593730 Fç$ytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16748415 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1459706289722400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚9599 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0FfZ£'† ŠN TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11138427554816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172593730 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16748415 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1459706289722400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚9599 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77649080 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1479431765296128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚330412842497536 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1470828243581475 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11121953859075 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11138768986112 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1479431406825472 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚330412712331776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319211330857984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39657119 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16748418 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39656832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11201400086016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1468292996310016 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1468292979270656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22908701 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1470827885980729 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94934491 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1479431765296128 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1468292979270656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11201511639552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172583571 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77649082 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1459706272683040 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172583571 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11138768986112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11138427554816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77649080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚789 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172593730 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94944648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94934491 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1468292996310016 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94944648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22908417 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360•1éÈ2‡ - _wandbruntime‚8698  _runtime‚8698„w¾ ˆ … 3loss/SRqqt2/assignment_loss‚0.44133223844440167 3loss/FBt2/assignment_loss‚0.0002008803244686713 /loss/FRt2/detection_loss‚1.4712650727538257 1loss/SRqqt2/detection_loss‚0.5938759269342243 _step‚790 .loss/FRt2/assignment_loss‚0.69628192751678 2loss/SRqqt1/assignment_loss‚0.4998069773384148 1loss/FBt1/assignment_loss‚0.04120895425691098 0loss/FRt1/detection_loss‚0.43758103392571984 1loss/SRqqt1/detection_loss‚0.7179439611852082 0loss/FBt1/detection_loss‚0.36981988516330455 _runtime‚8698.751741115 epoch‚23 trainer/global_step‚9599 0loss/FRt1/assignment_loss‚1.7064085799232653 /loss/FBt2/detection_loss‚0.5543550058540985 %loss/total_loss‚7.530080443620622 % _timestamp‚1.7845155251764607e+09l'‹8‰ Š _runtime‚8698.751741115 epoch‚23 trainer/global_step‚9599 0loss/FRt1/assignment_loss‚1.7064085799232653 /loss/FBt2/detection_loss‚0.5543550058540985 %loss/total_loss‚7.530080443620622 % _timestamp‚1.7845155251764607e+09 3loss/SRqqt2/assignment_loss‚0.44133223844440167 3loss/FBt2/assignment_loss‚0.0002008803244686713 /loss/FRt2/detection_loss‚1.4712650727538257 1loss/SRqqt2/detection_loss‚0.5938759269342243 _step‚790 .loss/FRt2/assignment_loss‚0.69628192751678 2loss/SRqqt1/assignment_loss‚0.4998069773384148 1loss/FBt1/assignment_loss‚0.04120895425691098 0loss/FRt1/detection_loss‚0.43758103392571984 1loss/SRqqt1/detection_loss‚0.7179439611852082 0loss/FBt1/detection_loss‚0.36981988516330455–€ÀøîuŠ ‚(:ê Æ—öÒ¾ØËË* proc.memory.percent‚18.447065353393555) disk./.usagePercent‚75.81245311792347& disk.nvme0c0n1.out‚50183.763671875( disk.nvme1c1n1.out‚25427.91357421875& disk.nvme1n1.out‚25427.91357421875 cpu‚25.688983181160676 network.recv‚ 2.308841e+06 disk.md0.in‚0 disk.dm-1.out‚11776.70703125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 190.734375& proc.memory.availableMB‚ 525.109375# disk./.usageGB‚1163.91068649292 disk.dm-0.in‚ 325.71484375# disk.dm-0.out‚63282.20556640625" disk.nvme0c0n1.in‚ 134.98046875 disk.nvme0n1.in‚ 134.98046875 disk.nvme1c1n1.in‚ 190.734375 proc.cpu.threads‚17 network.sent‚ 1.5273035e+07 disk.md0.out‚ 258.34375 disk.dm-1.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚50183.763671875$ memory_percent‚96.79498672485352" proc.memory.rssMB‚ 3022.3671875'ùCœ‹ ‚(:‘ Æ—öÒâ¦Ì gpu.0.gpu‚5.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.536' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.84533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔçF´‘Œ ‚(:† Õ—öÒÚÒ§Í gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.343' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.781 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÑ||x ‚(:í Õ—öÒ¦ìÝÍ# disk./.usageGB‚1161.19331741333& disk.nvme0c0n1.out‚50184.724609375# disk.dm-1.out‚11776.89208984375 disk.rbd0.out‚0 proc.cpu.threads‚17' disk.nvme1c1n1.out‚25428.6494140625* proc.memory.percent‚18.447065353393555 disk.nvme1n1.in‚ 190.734375% disk.nvme1n1.out‚25428.6494140625 disk.md0.in‚0 network.recv‚ 2.313001e+06" disk.nvme0c0n1.in‚ 134.98046875 disk.nvme0n1.in‚ 134.98046875$ disk.nvme0n1.out‚50184.724609375# disk.dm-0.out‚63282.78759765625 disk.md0.out‚ 258.76953125 disk.dm-0.in‚ 325.71484375 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚96.79665565490723' proc.memory.availableMB‚ 524.8359375" proc.memory.rssMB‚ 3022.3671875 cpu‚25.69291158265257 disk.nvme1c1n1.in‚ 190.734375 network.sent‚ 1.532466e+07) disk./.usagePercent‚75.63545464343463¥·wÕ2Ž - _wandbruntime‚8715  _runtime‚87152öyèo= éz =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873494505882263 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2405361831188202 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0472438782453537 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 *jet/accuracy_1_of_2‚0.7798293828964233 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26753684878349304 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15941156446933746 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2629414200782776 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5596536993980408 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6558739542961121 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6393641233444214 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5753868818283081 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6712440252304077 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5684280395507812 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.928911566734314 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6030397415161133 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13863767683506012 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9591553807258606 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6428830027580261 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982815980911255 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_3‚1 6validation_average_jet_accuracy‚0.3624860644340515 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.65625 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7073506712913513 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.041670460253953934 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6905114054679871 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6662920713424683 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3750000298023224 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6285777688026428 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6941407322883606 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2836482524871826 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34069952368736267 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956386089324951 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6086176633834839 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.30111947655677795 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7202971577644348 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9106301665306091 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6027827858924866 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47520509362220764 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.9180840253829956 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.634251594543457 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0920717716217041 % _timestamp‚1.7845155419621327e+09 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9180153608322144 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9875857830047607 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6305821537971497 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03915935009717941 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 epoch‚23 _step‚791 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1530952900648117 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5618888139724731 4 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2900370657444 'loss/val_total_loss‚7.5513862288742 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.583882212638855 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27138885855674744 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.537384569644928 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960317015647888 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4655475616455078 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6049911975860596 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6594178080558777 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07625389099121094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_3_of_4‚0.0926014706492424 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6001944541931152 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5596536993980408 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28344354033470154 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28591904044151306 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9321240186691284 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9212529063224792 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6442965269088745 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7202971577644348 /particle/accuracy_3_of_3‚0.9994120597839355 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6037819385528564 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_4‚0.9786046147346497 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4712183475494385 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.34732258319854736 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6752889156341553 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.9112623333930969 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6385080218315125 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1403764933347702 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948979616165161 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31877702474594116 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.550000011920929 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7618769407272339 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9097662568092346 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9893937706947327 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9746988415718079 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.569078803062439 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952585101127625 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9947980046272278 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9956252574920654 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2561970353126526 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6374932527542114 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6788628697395325 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15941156446933746 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22968009114265442 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7318119406700134 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27138885855674744 +particle/sensitivity‚0.6452111601829529 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.988358199596405 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5601880550384521 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15021635591983795 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2849646508693695 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2641800343990326 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3296944200992584 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23817408084869385 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8960262537002563 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2997835874557495 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8536242246627808 particle/accuracy_1_of_5‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32381507754325867 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26869863271713257 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33448004722595215 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974562525749207 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8738053441047668 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14787988364696503 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 'particle/f_score‚0.6543525457382202 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28114449977874756 *jet/accuracy_2_of_3‚0.5294876098632812 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988343119621277 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24928513169288635 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27216941118240356 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5977758169174194 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27839675545692444 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.33718013763427734 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20582658052444458 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.31897759437561035 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.670673131942749 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14675691723823547 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2168167233467102 +jet/accuracy_2_of_2‚0.14721238613128662 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2570253014564514 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 trainer/global_step‚9599 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5913856029510498 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8477891087532043 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6681703329086304 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9348639249801636 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053583934903144836 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964280128479004 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 particle/accuracy_1_of_1‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9484695196151733 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 _runtime‚8715.538381017 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5942408442497253 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7550029754638672 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963836669921875 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5230511426925659 +jet/accuracy_3_of_3‚0.07959591597318649 particle/accuracy_3_of_5‚1+p îz :"Purity/0FRt2SRqqt0FBt/event_purity‚0.34732258319854736 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6752889156341553 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.9112623333930969 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6385080218315125 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1403764933347702 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948979616165161 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31877702474594116 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.550000011920929 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7618769407272339 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9097662568092346 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9893937706947327 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9746988415718079 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.569078803062439 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt0SRqÔFç8qt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952585101127625 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9947980046272278 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9956252574920654 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2561970353126526 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6374932527542114 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6788628697395325 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15941156446933746 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22968009114265442 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7318119406700134 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27138885855674744 +particle/sensitivity‚0.6452111601829529 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.988358199596405 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5601880550384521 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8472222089767456 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15021635591983795 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2849646508693695 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2641800343990326 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3296944200992584 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23817408084869385 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8960262537002563 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2997835874557495 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8536242246627808 particle/accuracy_1_of_5‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32381507754325867 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26869863271713257 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33448004722595215 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974562525749207 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8738053441047668 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14787988364696503 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 'particle/f_score‚0.6543525457382202 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.28114449977874756 *jet/accuracy_2_of_3‚0.5294876098632812 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988343119621277 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24928513169288635 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27216941118240356 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5977758169174194 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27839675545692444 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.33718013763427734 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20582658052444458 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.31897759437561035 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.670673131942749 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14675691723823547 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2168167233467102 +jet/accuracy_2_of_2‚0.14721238613128662 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2570253014564514 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 trainer/global_step‚9599 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5913856029510498 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8477891087532043 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6681703329086304 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9348639249801636 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053583934903144836 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964280128479004 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 particle/accuracy_1_of_1‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9484695196151733 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 _runtime‚8715.538381017 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5942408442497253 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7550029754638672 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963836669921875 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5230511426925659 +jet/accuracy_3_of_3‚0.07959591597318649 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873494505882263 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2405361831188202 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0472438782453537 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8472222089767456 *jet/accuracy_1_of_2‚0.7798293828964233 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26753684878349304 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15941156446933746 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2629414200782776 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5596536993980408 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6558739542961121 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6393641233444214 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5753868818283081 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6712440252304077 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5684280395507812 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.928911566734314 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6030397415161133 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13863767683506012 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9591553807258606 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6428830027580261 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982815980911255 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_3‚1 6validation_average_jet_accuracy‚0.3624860644340515 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.65625 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7073506712913513 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.041670460253953934 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6905114054679871 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6662920713424683 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3750000298023224 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6285777688026428 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6941407322883606 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2836482524871826 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34069952368736267 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956386089324951 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6086176633834839 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.30111947655677795 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7202971577644348 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9106301665306091 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6027827858924866 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47520509362220764 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.9180840253829956 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.634251594543457 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0920717716217041 % _timestamp‚1.7845155419621327e+09 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9180153608322144 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9875857830047607 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6305821537971497 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03915935009717941 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 epoch‚23 _step‚791 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1530952900648117 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5618888139724731 4 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2900370657444 'loss/val_total_loss‚7.5513862288742 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.583882212638855 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27138885855674744 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.537384569644928 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960317015647888 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4655475616455078 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6049911975860596 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6594178080558777 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07625389099121094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_3_of_4‚0.0926014706492424 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6001944541931152 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5596536993980408 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28344354033470154 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28591904044151306 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9321240186691284 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9212529063224792 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6442965269088745 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7202971577644348 /particle/accuracy_3_of_3‚0.9994120597839355 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6037819385528564 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_4‚0.9786046147346497 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4712183475494385 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904— ©Þx‘ ‚(:í ä—öÒ†ÁËË disk.md0.out‚ 259.18359375 disk.nvme1c1n1.in‚ 190.734375$ memory_percent‚96.79415225982666 cpu‚25.692736851041452% disk./.usageGB‚1161.1933555603027 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 50195.7265625" disk.nvme0n1.out‚ 50195.7265625' disk.nvme1c1n1.out‚25430.2392578125( proc.memory.availableMB‚ 525.24609375" proc.memory.rssMB‚ 3022.3671875* proc.memory.percent‚18.447065353393555# disk.dm-1.out‚11776.97412109375 network.sent‚ 1.5340254e+07 disk.dm-0.in‚ 325.71484375# disk.dm-0.out‚63294.41455078125" disk.nvme0c0n1.in‚ 134.98046875 disk.nvme0n1.in‚ 134.98046875) disk./.usagePercent‚75.63545712817475 disk.dm-1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 2.31559e+06 disk.nvme1n1.in‚ 190.734375% disk.nvme1n1.out‚25430.2392578125 disk.md0.in‚0± ¸‚ž’ ‚(:“ ä—öÒëÞŽÌ gpu.0.gpu‚49.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.364' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.45466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÂ@Q“ ‚(:ô ó—öÒ»ÕÍË cpu‚25.692321169550418 disk.nvme0n1.in‚ 134.98046875) disk./.usagePercent‚75.61997645181974# disk.dm-0.out‚63295.04345703125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 134.98046875 disk.nvme1n1.in‚ 190.734375% disk./.usageGB‚1160.9556884765625' disk.nvme0c0n1.out‚50196.5791015625 disk.nvme1c1n1.in‚ 190.734375( disk.nvme1c1n1.out‚25430.83740234375& disk.nvme1n1.out‚25430.83740234375" proc.memory.rssMB‚ 3022.3671875 proc.cpu.threads‚17 network.recv‚ 2.319855e+06" disk.dm-1.out‚11777.0595703125 disk.md0.out‚ 259.51953125$ memory_percent‚96.79436683654785* proc.memory.percent‚18.447065353393555 network.sent‚ 1.5385405e+07 disk.dm-0.in‚ 325.71484375% disk.nvme0n1.out‚50196.5791015625 disk.md0.in‚0' proc.memory.availableMB‚ 525.2109375'áì´’” ‚(:‡ ó—öÒ™ùÌ gpu.0.gpu‚14.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚119.895' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.965 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊÃ2• - _wandbruntime‚8758  _runtime‚8758Ü@¡ŠÉ– à _runtime‚8758.816220423 _step‚792 ' lr-AdamW/pg1‚0.00033434444886158764 ' lr-AdamW/pg2‚0.00033434444886158764 trainer/global_step‚9649 % _timestamp‚1.7845155852395768e+09Þ1ÈΗ È ' lr-AdamW/pg2‚0.00033434444886158764 trainer/global_step‚9649 % _timestamp‚1.7845155852395768e+09 _runtime‚8758.816220423 _step‚792 ' lr-AdamW/pg1‚0.00033434444886158764˜Ÿ>f2˜ - _wandbruntime‚8758  _runtime‚87584G Ë(™ ûO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1480048280969768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1488713978144768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11213545817600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78153747 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332136627806720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320859742754304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11276903664640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1488713978144768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173654871 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39891977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1488714336608256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1468851648436752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16837497 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1477500773751296 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1488714336608256 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11213545817600 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95490981 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173644726 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78153747 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11213545817600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78153747 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332136627806720 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320859742754304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11276903664640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1488713978144768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173654871 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39891977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1488714336608256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1468851648436752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16837497 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1477500773751296 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1488714336608256 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11213545817600 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95490981 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173644726 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78153747 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 J2› - _wandbruntime‚8759  _runtime‚8759ƒƒ°z 'œ „N RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚9649 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1488863721216512 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1477649048253440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11214655923712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11214655923712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845155858633003e+09 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1488863362745856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16839231 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173662175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23056736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1480196820025604 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173672334 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1468999079104224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11197400359994 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78161540 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39895967 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1477649065292800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1477649065292800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1468999062064864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚320892923024384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚320892904412160 _step‚794 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173672334 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78161538 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1477649048253440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚332170922008576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚9649 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1488863721216512 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1477649048253440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11214655923712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11214655923712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845155858633003e+09 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1488863362745856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16839231 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173662175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23056736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1480196820025604 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173672334 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1468999079104224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11197400359994 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78161540 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39895967 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1477649065292800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1477649065292800 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1468999062064864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚320892923024384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚320892904412160 _step‚794 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173672334 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78161538 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1477649048253440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚332170922008576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78543395 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1487475297556776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 trainer/global_step‚9699 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95975633 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1476223164770992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11251775185150 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1476223181810352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40093889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78543397 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11252115746424 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78543395 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11269114919424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95975633 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11269114919424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11332847019520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78543397 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40094175 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1487474939956142 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1496183272353792 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11268773495296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚8802.097073059 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1496183630817280 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1484914515897856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚333851603277824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16924200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23169975 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16924197 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184öáE(¬ €P M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40093889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78543397 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11252115746424 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78543395 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11269114919424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95975633 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11269114919424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11332847019520 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78543397 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40094175 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1487474939956142 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1496183272353792 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11268773495296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚8802.097073059 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1496183630817280 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1484914515897856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚333851603277824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16924200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23169975 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16924197 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1496183630817280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚322518756258304 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174519028 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1484914515897856 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1484914498858496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1496183272353792 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1484914498858496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11268773495296 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 174529173 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333851481493504 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95985776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322518737646080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845156285207822e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78543395 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1487475297556776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 trainer/global_step‚9699 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95975633 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1476223164770992 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11251775185150 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1476223181810352ø Ç[2­ - _wandbruntime‚8802  _runtime‚8802Y?s} '® …N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174536477 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95985289 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1485062773360640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1487623479011866 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1496333015425536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174546636 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95985289 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78551188 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1485062790400000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40097856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333885778332672 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1476370595438464 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78551190 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78551190 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174546636 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95995446 _runtime‚8802.717902149 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚798 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚9699 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1485062790400000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322551917916160í½ZíÑ ¯ ŠN [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78551188 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1485062790400000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40097856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333885778332672 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1476370595438464 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78551190 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78551190 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174546636 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95995446 _runtime‚8802.717902149 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚798 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current¥èý7?‚1180 trainer/global_step‚9699 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1485062790400000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322551917916160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174536477 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95985289 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1485062773360640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1487623479011866 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1496333015425536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174546636 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95985289žˆ4½2° - _wandbruntime‚8802  _runtime‚8802k ± † 0loss/FRt1/assignment_loss‚1.6307212651424428 1loss/FBt1/assignment_loss‚0.03363020395603419 0loss/FRt1/detection_loss‚0.47438285842577144 /loss/FRt2/detection_loss‚1.5343059248416446 1loss/SRqqt1/detection_loss‚0.8015019175786031 0loss/FBt2/detection_loss‚0.42776147107360885 epoch‚24 $ _timestamp‚1.784515629143633e+09 0loss/FRt2/assignment_loss‚0.8054177026971677 2loss/FBt2/assignment_loss‚0.025024409275741566 1loss/SRqqt2/detection_loss‚0.7100217126781126 /loss/FBt1/detection_loss‚0.5052879398538678 %loss/total_loss‚7.908821506476807 3loss/SRqqt2/assignment_loss‚0.46218628406276513 _runtime‚8802.719041952 3loss/SRqqt1/assignment_loss‚0.49857981689104747 trainer/global_step‚9699 _step‚799õϲ ‹ epoch‚24 $ _timestamp‚1.784515629143633e+09 0loss/FRt2/assignment_loss‚0.8054177026971677 2loss/FBt2/assignment_loss‚0.025024409275741566 1loss/SRqqt2/detection_loss‚0.7100217126781126 /loss/FBt1/detection_loss‚0.5052879398538678 %loss/total_loss‚7.908821506476807 3loss/SRqqt2/assignment_loss‚0.46218628406276513 _runtime‚8802.719041952 3loss/SRqqt1/assignment_loss‚0.49857981689104747 trainer/global_step‚9699 _step‚799 0loss/FRt1/assignment_loss‚1.6307212651424428 1loss/FBt1/assignment_loss‚0.03363020395603419 0loss/FRt1/detection_loss‚0.47438285842577144 /loss/FRt2/detection_loss‚1.5343059248416446 1loss/SRqqt1/detection_loss‚0.8015019175786031 0loss/FBt2/detection_loss‚0.42776147107360885ŸŠ‡}³ ‚(:ò ¯˜öÒ¶£ÓË network.recv‚ 2.334302e+06 disk.md0.in‚0" disk.dm-0.out‚63320.4443359375# disk.dm-1.out‚11777.42333984375$ memory_percent‚96.79791927337646" proc.memory.rssMB‚ 3022.3671875) disk./.usagePercent‚75.61996328269711 disk.dm-0.in‚ 325.71484375 disk.dm-1.in‚0$ disk.nvme1n1.out‚25437.123046875 cpu‚25.69072935833966( disk.nvme0c0n1.out‚50219.48779296875 network.sent‚ 1.5474461e+07 disk.md0.out‚ 261.09765625 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚50219.48779296875 disk.nvme1n1.in‚ 190.734375( proc.memory.availableMB‚ 524.62890625% disk./.usageGB‚1160.9554862976074 disk.nvme0n1.in‚ 134.98046875 disk.nvme1c1n1.in‚ 190.734375& disk.nvme1c1n1.out‚25437.123046875* proc.memory.percent‚18.447065353393555 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 134.98046875ÆF™Ãž´ ‚(:“ ¯˜öÒÖ¼Ì gpu.0.gpu‚88.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚205.249' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚68.41633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0– ò|µ ‚(:ñ ¾˜öÒ¼®±Ë disk.md0.out‚ 261.578125 disk.rbd0.out‚0 disk.dm-0.in‚ 325.71484375& proc.memory.availableMB‚ 524.609375" proc.memory.rssMB‚ 3022.3671875 cpu‚25.690322327269314 proc.cpu.threads‚17 disk.nvme1n1.in‚ 190.734375) disk./.usagePercent‚75.61993868376994% disk./.usageGB‚1160.9551086425781" disk.nvme0c0n1.in‚ 134.98046875 disk.nvme1c1n1.in‚ 190.734375# disk.dm-1.out‚11777.56005859375* proc.memory.percent‚18.447065353393555 network.recv‚ 2.338378e+06 disk.nvme0n1.in‚ 134.98046875" disk.dm-0.out‚63322.2490234375 disk.dm-1.in‚0$ memory_percent‚96.79803848266602' disk.nvme1c1n1.out‚25437.9853515625% disk.nvme1n1.out‚25437.9853515625 network.sent‚ 1.5523091e+07 disk.rbd0.in‚0( disk.nvme0c0n1.out‚50221.58837890625& disk.nvme0n1.out‚50221.58837890625 disk.md0.in‚0ÖZ架 ‚(:“ ¾˜öÒ…Ò†Ì gpu.0.gpu‚97.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚268.756' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.58533333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t;`¹€· ‚(:õ ͘öÒé Ë disk.rbd0.out‚0' disk.nvme0c0n1.out‚50223.5224609375% disk.nvme0n1.out‚50223.5224609375 disk.md0.in‚0* proc.memory.percent‚18.447065353393555 disk.nvme1n1.in‚ 190.734375 disk.dm-0.in‚ 325.71484375 disk.dm-1.in‚0# disk.dm-1.out‚11777.63720703125$ memory_percent‚96.79820537567139( proc.memory.availableMB‚ 524.58203125 proc.cpu.threads‚17 disk.rbd0.in‚0& disk.nvme1n1.out‚25439.48095703125 disk.md0.out‚ 262.86328125" proc.memory.rssMB‚ 3022.3671875 network.sent‚ 1.5529035e+07% disk./.usageGB‚1160.9552841186523" disk.nvme0c0n1.in‚ 134.98046875 disk.nvme0n1.in‚ 134.98046875 disk.nvme1c1n1.in‚ 190.734375( disk.nvme1c1n1.out‚25439.48095703125" disk.dm-0.out‚63322.9443359375 cpu‚25.690085312188778 network.recv‚ 2.342059e+06) disk./.usagePercent‚75.61995011357449x´ŸÑŸ¸ ‚(:” ͘öÒ¶´êË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚124.934' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.644666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…8>š2¹ - _wandbruntime‚8845  _runtime‚8845ˆ†LǺ Á & lr-AdamW/pg1‚0.0003294356891646229 & lr-AdamW/pg2‚0.0003294356891646229 trainer/global_step‚9749 % _timestamp‚1.7845156721659544e+09 _runtime‚8845.741989408 _step‚800 hRjÌ» Æ & lr-AdamW/pg2‚0.0003294356891646229 trainer/global_step‚9749 % _timestamp‚1.7845156721659544e+09 _runtime‚8845.741989408 _step‚800 & lr-AdamW/pg1‚0.0003294356891646229 ¡•E2¼ - _wandbruntime‚8845  _runtime‚8845ñXËy(½ ûO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11324684021248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78933045 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1492328241005056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1503652566562816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 % _timestamp‚1.7845156721667418e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1492328223965696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1492328241005056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚9749 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1492328223965696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78933047 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1503652925026304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11388591308800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1503652566562816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _step‚801 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚324177769762304 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1503652925026304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78933047 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23285396 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11307258398558 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96460285 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11324684021248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11324342597120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175403475 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23285108 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324177751150080 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11388698675200 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1494902314143784 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11307598959832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0U‹ú¶&¾ €P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11324684021248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78933045 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1492328241005056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1503652566562816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 % _timestamp‚1.7845156721667418e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1492328223965696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1492328241005056 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚9749 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1492328223965696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78933047 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1503652925026304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11388591308800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1503652566562816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _step‚801 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚324177769762304 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1503652925026304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78933047 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23285396 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11307258398558 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96460285 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11324684021248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11324342597120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175403475 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23285108 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚324177751150080 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11388698675200 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1494902314143784 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11307598959832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive>I†ÙP_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374¡ʬ92¿ - _wandbruntime‚8846  _runtime‚8846 öU8 'À …N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175410779 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175420938 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11389708050432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1495050495598874 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1503802309634560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1492476498467840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78940838 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335600761247744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11325794127360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1503802309634560 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1492476498467840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78940840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78940838 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1483742145851424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40300402 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17012634 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175410779 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17012631 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1495050853199620 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1503801951163904 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40300109 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23287768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Å ÛŒ'Á ŠN UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175410779 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175420938 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11389708050432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1495050495598874 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1503802309634560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1492476498467840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78940838 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335600761247744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11325794127360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1503802309634560 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1492476498467840 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78940840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78940838 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1483742145851424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40300402 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17012634 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175410779 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17012631 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1495050853199620 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1503801951163904 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40300109 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23287768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647¢n'=:2 - _wandbruntime‚8846  _runtime‚8846`[ü à ƒ 0loss/FRt1/assignment_loss‚1.7084073603632943 2loss/SRqqt1/assignment_loss‚0.5262733618287376 .loss/FRt2/detection_loss‚1.452829139012747 /loss/FBt1/detection_loss‚0.4676430687126329 0loss/FBt2/detection_loss‚0.43669197109635344 %loss/total_loss‚7.418727448221357 _step‚803 /loss/FRt2/assignment_loss‚0.565581149893774 /loss/FRt1/detection_loss‚0.4433472112420625 1loss/SRqqt2/detection_loss‚0.6056217186809909 trainer/global_step‚9749 _runtime‚ 8846.3657703 4loss/FBt2/assignment_loss‚0.00015522669205005548 1loss/SRqqt1/detection_loss‚0.7682992006080137 2loss/SRqqt2/assignment_loss‚0.4412113772081778 3loss/FBt1/assignment_loss‚0.0026666628825221055 epoch‚24 $ _timestamp‚1.784515672790477e+09ÁÌ2Ä ˆ /loss/FRt2/assignment_loss‚0.565581149893774 /loss/FRt1/detection_loss‚0.4433472112420625 1loss/SRqqt2/detection_loss‚0.6056217186809909 trainer/global_step‚9749 _runtime‚ 8846.3657703 4loss/FBt2/assignment_loss‚0.00015522669205005548 1loss/SRqqt1/detection_loss‚0.7682992006080137 2loss/SRqqt2/assignment_loss‚0.4412113772081778 3loss/FBt1/assignment_loss‚0.0026666628825221055 epoch‚24 $ _timestamp‚1.784515672790477e+09 0loss/FRt1/assignment_loss‚1.7084073603632943 2loss/SRqqt1/assignment_loss‚0.5262733618287376 .loss/FRt2/detection_loss‚1.452829139012747 /loss/FBt1/detection_loss‚0.4676430687126329 0loss/FBt2/detection_loss‚0.43669197109635344 %loss/total_loss‚7.418727448221357 _step‚803£Ó"8Å ‚(:ô ܘöÒ¿¹ÙË disk.md0.in‚0 disk.dm-0.in‚ 325.71484375# disk.dm-0.out‚63325.17822265625 disk.dm-1.in‚0# disk.dm-1.out‚11777.68408203125" disk.nvme0c0n1.in‚ 134.98046875 network.sent‚ 1.5546721e+07 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 134.98046875( proc.memory.availableMB‚ 524.54296875* proc.memory.percent‚18.447065353393555 cpu‚25.689704645597867 proc.cpu.threads‚17 network.recv‚ 2.345148e+06' disk.nvme0c0n1.out‚50225.9501953125 disk.nvme1c1n1.in‚ 190.734375) disk./.usagePercent‚75.61995110747054% disk./.usageGB‚1160.9552993774414 disk.md0.out‚ 263.26171875% disk.nvme0n1.out‚50225.9501953125' disk.nvme1c1n1.out‚25440.1748046875$ memory_percent‚96.79844379425049" proc.memory.rssMB‚ 3022.3671875 disk.nvme1n1.in‚ 190.734375% disk.nvme1n1.out‚25440.17480468751³GaœÆ ‚(:‘ ܘöÒ‘Ù¢Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.538' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.17933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tÈ%yÇ ‚(:î ë˜öÒß„¡Ë) disk./.usagePercent‚75.61997993045591% disk.nvme1n1.out‚25566.4384765625 disk.md0.out‚ 389.265625 disk.nvme0n1.in‚ 134.98046875 disk.nvme1c1n1.in‚ 190.734375 proc.cpu.threads‚17 disk.nvme1n1.in‚ 190.734375 disk.dm-0.in‚ 325.71484375# disk.dm-0.out‚63325.98681640625 disk.dm-1.in‚0# disk.dm-1.out‚11777.80908203125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 134.98046875 cpu‚25.68938691388333% disk./.usageGB‚1160.9557418823242 disk.rbd0.in‚0$ memory_percent‚96.79851531982422% proc.memory.availableMB‚ 524.53125" proc.memory.rssMB‚ 3022.3671875* proc.memory.percent‚18.447065353393555 network.recv‚ 2.349522e+06 disk.md0.in‚0' disk.nvme0c0n1.out‚50352.7490234375% disk.nvme0n1.out‚50352.7490234375' disk.nvme1c1n1.out‚25566.4384765625 network.sent‚ 1.5595414e+07¢å+*žÈ ‚(:“ ë˜öÒºÛòË gpu.0.gpu‚93.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.356' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.78533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à» sÉ ‚(:è ú˜öÒÁʺË( disk./.usagePercent‚75.6200008022729 disk.rbd0.in‚0% disk.nvme1c1n1.out‚25567.41796875 disk.md0.in‚0 disk.md0.out‚ 389.90625" proc.memory.rssMB‚ 3022.3671875 network.recv‚ 2.352954e+06! disk.dm-0.out‚63328.994140625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 134.98046875 disk.nvme1n1.in‚ 190.734375# disk.nvme1n1.out‚25567.41796875* proc.memory.percent‚18.447065353393555 cpu‚25.68901016903893 network.sent‚ 1.5601401e+07 disk.dm-0.in‚ 325.71484375# disk.dm-1.out‚11777.85205078125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 134.98046875( disk.nvme0c0n1.out‚50356.16162109375& disk.nvme0n1.out‚50356.16162109375$ memory_percent‚96.79851531982422% disk./.usageGB‚1160.9560623168945 disk.nvme1c1n1.in‚ 190.734375% proc.memory.availableMB‚ 524.53125 proc.cpu.threads‚17A"LžÊ ‚(:“ ú˜öÒ—©‘Ì gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚268.194' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.39800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jc2Ë - _wandbruntime‚8889  _runtime‚8889ÇêÉÌ Ã _step‚804 ' lr-AdamW/pg1‚0.00032697909254301045 ' lr-AdamW/pg2‚0.00032697909254301045 trainer/global_step‚9799 % _timestamp‚1.7845157154386027e+09 _runtime‚8889.014192616¬ÝÎÍ È trainer/global_step‚9799 % _timestamp‚1.7845157154386027e+09 _runtime‚8889.014192616 _step‚804 ' lr-AdamW/pg1‚0.00032697909254301045 ' lr-AdamW/pg2‚0.00032697909254301045¤ŽÓ·¼2Î - _wandbruntime‚8889  _runtime‚8889¯l?C^Ï ûO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1490966248557552 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1511122219235328 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚9799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23400815 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79322697 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337281287570432 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1502328973130158 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1511121860771840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11363082173240 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1499741966112256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96944937 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚337281159494656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96955080 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11380253123072 % _timestamp‚1.7845157154396367e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚805 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1499741949072896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11362741611966 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176267632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79322695 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚325836783266304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23400527 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11379911698944 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1490966231518192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚8889.015274477 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79322695 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1499741966112256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40498415 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11444394840576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed¶f#£‚17097597 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1499741949072896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96955080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11379911698944 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1511122219235328 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11380253123072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11444504304128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1511121860771840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176277777 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79322697 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96944937 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40498124 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚325836764654080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1502329330730792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1511121860771840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176277777 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79322697 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96944937 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40498124 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚325836764654080 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1502329330730792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1511121860771840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11363082173240 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1499741966112256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96944937 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚337281159494656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96955080 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11380253123072 % _timestamp‚1.7845157154396367e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚805 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1499741949072896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11362741611966 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176267632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79322695 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚325836783266304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23400527 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11379911698944 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1490966231518192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚8889.015274477 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79322695 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1499741966112256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40498415¥£…Ÿ2Ñ - _wandbruntime‚8889  _runtime‚8889É`â 'Ò …N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1499890223575040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1502477869786628 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79330490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337315450911232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23403118 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚325869944924160 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1491113679225024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96964750 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11363850000218 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11381021797888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11381021797888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337315585271296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11445505987072 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1511271245372928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11445621734912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 _step‚806 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96964750 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11381363229184 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96954593 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176295240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40502452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7845157160638263e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134360064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11364190561604 trainer/global_step‚9799 _runtime‚8889.639588336 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176295240 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1491113662185664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1502477512185882 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176285081 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40502167 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23402836 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79330490 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176295240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40502452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7845157160638263e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134360064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11364190561604 trainer/global_step‚9799 _runtime‚8889.639588336 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176295240 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1491113662185664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1502477512185882 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176285081 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40502167 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23402836 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79330490 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79330490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337315450911232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inact[Ž•úive_split.small_pool.allocated‚23403118 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚325869944924160 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1491113679225024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96964750 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11363850000218 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11381021797888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11381021797888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337315585271296¦Š:.@2Ô - _wandbruntime‚8889  _runtime‚8889ÆíÇçÕ € 0loss/FRt2/assignment_loss‚0.6482781225643238 2loss/SRqqt2/assignment_loss‚0.5299923590792714 2loss/FBt2/assignment_loss‚0.005147684888321362 1loss/SRqqt2/detection_loss‚0.5972741305517654 /loss/FBt1/detection_loss‚0.4863586422987282 /loss/FBt2/detection_loss‚0.5144011275021908 epoch‚24 trainer/global_step‚9799 2loss/SRqqt1/assignment_loss‚0.4074084352540712 $loss/total_loss‚7.62034354070312 $ _timestamp‚1.784515716064901e+09 0loss/FRt1/assignment_loss‚1.6879067842059228 /loss/FRt1/detection_loss‚0.4561589419170224 2loss/FBt1/assignment_loss‚0.024745661606494666 -loss/FRt2/detection_loss‚1.38040256680052 1loss/SRqqt1/detection_loss‚0.8822690840344876 _runtime‚8889.640192565 _step‚807åÙ Ö … 0loss/FRt2/assignment_loss‚0.6482781225643238 2loss/SRqqt2/assignment_loss‚0.5299923590792714 2loss/FBt2/assignment_loss‚0.005147684888321362 1loss/SRqqt2/detection_loss‚0.5972741305517654 /loss/FBt1/detection_loss‚0.4863586422987282 /loss/FBt2/detection_loss‚0.5144011275021908 epoch‚24 trainer/global_step‚9799 2loss/SRqqt1/assignment_loss‚0.4074084352540712 $loss/total_loss‚7.62034354070312 $ _timestamp‚1.784515716064901e+09 0loss/FRt1/assignment_loss‚1.6879067842059228 /loss/FRt1/detection_loss‚0.4561589419170224 2loss/FBt1/assignment_loss‚0.024745661606494666 -loss/FRt2/detection_loss‚1.38040256680052 1loss/SRqqt1/detection_loss‚0.8822690840344876 _runtime‚8889.640192565 _step‚807§Cþ~× ‚(:ó ‰™öÒ¹çÙË" proc.memory.rssMB‚ 3022.3671875 network.recv‚ 2.356506e+06 disk.dm-1.in‚0" disk.dm-1.out‚11778.0263671875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 134.98046875( disk.nvme0c0n1.out‚50468.81298828125 disk.nvme1c1n1.in‚ 190.734375& disk.nvme1n1.out‚25680.18115234375 disk.nvme1n1.in‚ 190.734375$ memory_percent‚96.79892063140869 cpu‚25.68866248103978 proc.cpu.threads‚17% disk./.usageGB‚1160.9560623168945 disk.dm-0.in‚ 325.71484375! disk.dm-0.out‚63329.720703125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 134.98046875( disk.nvme1c1n1.out‚25680.18115234375* proc.memory.percent‚18.447065353393555 network.sent‚ 1.5619074e+07( disk./.usagePercent‚75.6200008022729& disk.nvme0n1.out‚50468.81298828125 disk.md0.in‚0 disk.md0.out‚ 502.12890625( proc.memory.availableMB‚ 524.46484375«?:_žØ ‚(:“ ‰™öÒƒ¸˜Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.524' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.84133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+§ÏÔ|Ù ‚(:ñ ˜™öÒïÈË proc.cpu.threads‚17 network.sent‚ 1.566942e+07 network.recv‚ 2.36075e+06) disk./.usagePercent‚75.75081839388307% disk./.usageGB‚1162.9644393920898" disk.nvme0c0n1.in‚ 134.98046875& disk.nvme1n1.out‚25680.93408203125 disk.md0.in‚0# disk.dm-0.out‚63333.26904296875' disk.nvme0c0n1.out‚50472.4814453125 disk.nvme1c1n1.in‚ 190.734375 disk.md0.out‚ 502.51171875" disk.dm-1.out‚11778.0732421875 disk.rbd0.in‚0# memory_percent‚96.7989444732666" proc.memory.rssMB‚ 3022.3671875* proc.memory.percent‚18.447065353393555( disk.nvme1c1n1.out‚25680.93408203125 disk.dm-1.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 524.4609375 cpu‚25.688290290953923 disk.nvme0n1.in‚ 134.98046875% disk.nvme0n1.out‚50472.4814453125 disk.nvme1n1.in‚ 190.734375 disk.dm-0.in‚ 325.71484375*ªÕB‘Ú ‚(:† ˜™öÒëûŒÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.714' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|½—}Û ‚(:ò §™öÒ™ÖÙË proc.cpu.threads‚17 network.recv‚ 2.364255e+06% disk./.usageGB‚1166.9309005737305& disk.nvme0n1.out‚53246.04443359375( disk.nvme1c1n1.out‚27735.61669921875 disk.md0.in‚0 disk.dm-0.in‚ 325.71484375 disk.dm-1.in‚0* proc.memory.percent‚18.447065353393555 network.sent‚ 1.5674347e+07" disk.nvme0c0n1.in‚ 134.98046875 disk.nvme0n1.in‚ 134.98046875 disk.nvme1n1.in‚ 190.734375& disk.nvme1n1.out‚27735.61669921875" disk.dm-0.out‚68160.0400390625" disk.dm-1.out‚11778.2216796875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚53246.04443359375 disk.nvme1c1n1.in‚ 190.734375# memory_percent‚96.7989444732666) disk./.usagePercent‚76.00917769573232 disk.md0.out‚ 503.140625 disk.rbd0.in‚0' proc.memory.availableMB‚ 524.4609375" proc.memory.rssMB‚ 3022.3671875 cpu‚25.687945603000774L#7“Ü ‚(:ˆ §™öÒ±õ•Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚284.928' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m£r2Ý - _wandbruntime‚8933  _runtime‚8933ü öÉÞ Ã ' lr-AdamW/pg2‚0.00032452122841894433 trainer/global_step‚9849 % _timestamp‚1.7845157594559147e+09 _runtime‚8933.032061255 _step‚808 ' lr-AdamW/pg1‚0.00032452122841894433YܹdÎß È _runtime‚8933.032061255 _step‚808 ' lr-AdamW/pg1‚0.00032452122841894433 ' lr-AdamW/pg2‚0.00032452122841894433 trainer/global_step‚9849 % _timestamp‚1.7845157594559147e+09¨Jû^~2à - _wandbruntime‚8933  _runtime‚8933Éø°(á ûO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1518591154980864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23516137 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1498337764891792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327495778158080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11435480800768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97429589 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1518591513444352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40700153 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11500220595712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97429589 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17184300 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚9849 % _timestamp‚1.7845157594566603e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1518591513444352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79712347 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327495796770304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17184297 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚338995998753792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11435822224896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97439732 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚809 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1498337781931152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177141934 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1518591154980864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1507155674180096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177152079 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1507155691219456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11500321670656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1507155691219456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79712347 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79712345 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11418565386648 _runtime‚8933.032265165 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1507155674180096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40700437 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97439732 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23515856 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11418224825374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274Àí z(â €P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1507155691219456 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79712347 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79712345 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11418565386648 _runtime‚8933.032265165 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1507155674180096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40700437 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97439732 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23515856 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11418224825374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1518591154980864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23516137 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1498337764891792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327495778158080 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11435480800768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97429589 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1518591513444352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40700153 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11500220595712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97429589 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17184300 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚9849 % _timestamp‚1.7845157594566603e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1518591513444352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79712347 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327495796770304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17184297 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚338995998753792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11435822224896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97439732 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚809 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1498337781931152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177141934 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1518591154980864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1507155674180096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177152079 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1507155691219456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11500321670656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634©>ŒÇ42ã - _wandbruntime‚8933  _runtime‚8933͈sç¹ä „N MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327528958428160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1509904528772890 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177169542 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1518740898052608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177159383 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17186034 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1509904886373636 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97439245 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11419333213626 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40704209 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11501434001920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peakqN™¨Q‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177169542 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1498485195559264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97449402 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845157600822816e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1507303965721600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79720138 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1507303965721600 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1518740898052608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79720140 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339030411042304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327528977040384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1507303948682240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177159383 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11436590899712 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1518740539581952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚8933.658037268 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11436932331008 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11436590899712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339030297653760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79720138 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177169542 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1498485195559264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97449402 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845157600822816e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1507303965721600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79720138 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1507303965721600 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1518740898052608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79720140 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339030411042304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327528977040384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1507303948682240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177159383 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11436590899712 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1518740539581952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚8933.658037268 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11436932331008 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11436590899712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339030297653760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79720138 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177159383 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17186034 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1509904886373636 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97439245 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11419333213626 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40704209 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11501434001920 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272ª¢lë2æ - _wandbruntime‚8933  _runtime‚8933N°k£ ç ƒ 3loss/FBt2/assignment_loss‚0.0004085622282668942 0loss/SRqqt1/detection_loss‚0.794798535593411 1loss/SRqqt2/detection_loss‚0.5902393109313033 %loss/total_loss‚7.494726220066709 0loss/FRt1/assignment_loss‚1.7269926778289306 1loss/SRqqt2/assignment_loss‚0.512473110502989 /loss/FRt2/detection_loss‚1.4853622824471948 $ _timestamp‚1.784515760083333e+09 _runtime‚8933.658733665 _step‚811 /loss/FRt2/assignment_loss‚0.684328453768119 3loss/FBt1/assignment_loss‚0.0007874732510738208 0loss/FRt1/detection_loss‚0.44825904224865215 epoch‚24 trainer/global_step‚9849 2loss/SRqqt1/assignment_loss‚0.5273386847539735 /loss/FBt1/detection_loss‚0.3513627000076401 /loss/FBt2/detection_loss‚0.3723753865051549s@‚è ˆ /loss/FBt2/detection_loss‚0.3723753865051549 3loss/FBt2/assignment_loss‚0.0004085622282668942 0loss/SRqqt1/detection_loss‚0.794798535593411 1loss/SRqqt2/detection_loss‚0.5902393109313033 %loss/total_loss‚7.494726220066709 0loss/FRt1/assignment_loss‚1.7269926778289306 1loss/SRqqt2/assignment_loss‚0.512473110502989 /loss/FRt2/detection_loss‚1.4853622824471948 $ _timestamp‚1.784515760083333e+09 _runtime‚8933.658733665 _step‚811 /loss/FRt2/assignment_loss‚0.684328453768119 3loss/FBt1/assignment_loss‚0.0007874732510738208 0loss/FRt1/detection_loss‚0.44825904224865215 epoch‚24 trainer/global_step‚9849 2loss/SRqqt1/assignment_loss‚0.5273386847539735 /loss/FBt1/detection_loss‚0.3513627000076401«V Ï}é ‚(:ò ¶™öÒ­˜Ë) disk./.usagePercent‚76.00472181127934 disk.md0.in‚0 disk.dm-0.in‚ 325.71484375" disk.dm-1.out‚11778.3193359375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 134.98046875& disk.nvme0n1.out‚54034.19189453125' disk.nvme1c1n1.out‚28193.0810546875 network.sent‚ 1.5677412e+07 disk.md0.out‚ 503.3984375 disk.rbd0.in‚0% disk.nvme1n1.out‚28193.0810546875* proc.memory.percent‚18.447065353393555 proc.cpu.threads‚17 network.recv‚ 2.367626e+06$ disk./.usageGB‚1166.862491607666( disk.nvme0c0n1.out‚54034.19189453125$ memory_percent‚96.80287837982178 cpu‚25.68760599464702# disk.dm-0.out‚69404.98486328125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 134.98046875 disk.nvme1c1n1.in‚ 190.734375 disk.nvme1n1.in‚ 190.734375( proc.memory.availableMB‚ 523.81640625" proc.memory.rssMB‚ 3022.3671875½$„“ê ‚(:ˆ ¶™öÒÃζÍ gpu.0.gpu‚97.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚288.459' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.153 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A"zë ‚(:ï Å™öÒýâ¾Ë disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.out‚ 503.86328125" disk.nvme0c0n1.in‚ 134.98046875( disk.nvme0c0n1.out‚54036.51123046875& disk.nvme0n1.out‚54036.51123046875 disk.nvme1n1.in‚ 190.734375$ proc.memory.availableMB‚524.0625" proc.memory.rssMB‚ 3022.3671875* proc.memory.percent‚18.447065353393555 cpu‚25.687234807030325) disk./.usagePercent‚76.00471311468893# disk.dm-1.out‚11779.25927734375 disk.nvme0n1.in‚ 134.98046875$ disk.nvme1n1.out‚28194.603515625 disk.md0.in‚0 network.recv‚ 2.372013e+06 disk.dm-0.in‚ 325.71484375" disk.dm-0.out‚69406.8935546875$ memory_percent‚96.80137634277344 proc.cpu.threads‚17 disk.nvme1c1n1.in‚ 190.734375& disk.nvme1c1n1.out‚28194.603515625 network.sent‚ 1.5739783e+07% disk./.usageGB‚1166.8623580932617€sLv“ì ‚(:ˆ Å™öÒìÁ‘Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.155' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.385 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â°àçtí ‚(:é Ô™öÒŠªÒË network.sent‚ 1.5746343e+07) disk./.usagePercent‚76.00470590894258" disk.nvme0c0n1.in‚ 134.98046875' disk.nvme0c0n1.out‚54046.8291015625 disk.dm-1.in‚0$ disk./.usageGB‚1166.862247467041 disk.nvme0n1.in‚ 134.98046875% disk.nvme0n1.out‚54046.8291015625 disk.nvme1c1n1.in‚ 190.734375% disk.nvme1c1n1.out‚28197.73046875 disk.nvme1n1.in‚ 190.734375 disk.md0.in‚0# disk.dm-0.out‚69419.62744140625# disk.nvme1n1.out‚28197.73046875 disk.dm-0.in‚ 325.71484375 disk.rbd0.in‚0 disk.rbd0.out‚0$ proc.memory.availableMB‚523.8125 proc.cpu.threads‚17 network.recv‚ 2.374587e+06 disk.md0.out‚ 504.1640625# disk.dm-1.out‚11779.31787109375$ memory_percent‚96.80290222167969" proc.memory.rssMB‚ 3022.3671875* proc.memory.percent‚18.447065353393555 cpu‚25.68689349438291õø“î ‚(:ˆ Ô™öÒ¤ ’Ì gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚263.019' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚87.673 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P¹Æú2ï - _wandbruntime‚8977  _runtime‚8977wŒNÇð Á trainer/global_step‚9899 % _timestamp‚1.7845158036465652e+09 _runtime‚8977.222166985 _step‚812 & lr-AdamW/pg1‚0.0003220622546564443 & lr-AdamW/pg2‚0.0003220622546564443ÞÏò¾Ìñ Æ trainer/global_step‚9899 % _timestamp‚1.7845158036465652e+09 _runtime‚8977.222166985 _step‚812 & lr-AdamW/pg1‚0.0003220622546564443 & lr-AdamW/pg2‚0.0003220622546564443¬¯‘52ò - _wandbruntime‚8977  _runtime‚8977*ØÑÌ(ó ûO ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1505709298265392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1526060807653376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23632471 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178016236 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1517183363904808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97914241 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1514569416326656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178026381 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97924384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17270997 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1514569399287296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340710863674880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329154810274304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80101997 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40903186 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1526060449189888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845158036475897e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11474048600056 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97924384 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1526060449189888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80101997 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80101995 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40903471 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11556173087744 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11556072012800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11491049902592 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1514569399287296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11491391326720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340710983362048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚9899fiéÄô €P TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23632189 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_start/requested_by-³öÕB$tes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1505709315304752 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1526060449189888 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845158036475897e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11474048600056 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97924384 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1526060449189888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80101997 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80101995 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40903471 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11556173087744 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11556072012800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11491049902592 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1514569399287296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11491391326720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340710983362048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚9899 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1505709298265392 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1526060807653376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23632471 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178016236 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1517183363904808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97914241 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1514569416326656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178026381 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97924384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17270997 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1514569399287296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340710863674880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329154810274304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80101997 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40903186 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178043844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17272731 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚329187971932160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1514717673789440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178033685 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 trainer/global_step‚9899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178033685 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97934054 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17272734 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1526210192261632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11492501432832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329187990544384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11475156988420 _step‚814 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11492501432832 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1526210192261632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80109790 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23634735 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40907175 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1517331902960644 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80109788 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97923897 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97923897 _runtime‚8977.849696514 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚340745279764480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11474816427034 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80109788 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340745155890176 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1505856728932864 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0‚MdÓ'÷ ŠN K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340745155890176 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1505856728932864 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1526209833790976 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1514717690828800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178043844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17272731 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚329187971932160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1514717673789440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178033685 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 trainer/global_step‚9899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178033685 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97934054 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17272734 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1526210192261632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11492501432832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329187990544384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11475156988420 _step‚814 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11492501432832 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1526210192261632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80109790 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23634735 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40907175 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1517331902960644 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80109788 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97923897 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97923897 _runtime‚8977.849696514 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚340745279764480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11474816427034 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80109788 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80491645 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1524610022891182 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚817 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1533530101862400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11546960428544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17357700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚9021.754954749 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1521983141433856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚342425662949888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11611958858752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23747334 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1533530101862400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178890538 trainer/global_step‚9949 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41105031 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11546619004416 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1521983124394496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178900683 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330813805166080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1524610380491816 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23747612 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41105312 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 % _timestamp‚1.7845158481778085e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11529531813464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚342425782637056 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1533529743398912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1513080848678352 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98398893 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11546619004416 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1533529743398912 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚342425782637056 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1533529743398912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1513080848678352 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98398893 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11546619004416 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1533529743398912 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80491645 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1524610022891182 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚817 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1533530101862400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11546960428544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17357700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚9021.754954749 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1521983141433856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚342425662949888 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11611958858752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23747334 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1533530101862400 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178890538 trainer/global_step‚9949 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41105031 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11546619004416 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1521983124394496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178900683 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330813805166080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1524610380491816 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23747612 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41105312±^3ª2‡!- _wandbruntime‚9022  _runtime‚9022kŠÅ%ˆ!ƒN YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11530299640442 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1524758919547652 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178918146 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11612969774080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11548070534656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚342460079084544 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _runtime‚ 9022.37937293 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330847004048384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178907987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80499440 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11548070534656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330846985436160 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1513228262306464 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1522131415936000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98408549 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17359431 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41109346 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80499440 _step‚818 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98418706 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98408549 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17359434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178907987 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178918146 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11530640201828 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1522131398896640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1524758561946906 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80499438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 $ _timestamp‚1.784515848803579e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80499438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41109057 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23749626 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.Þo®ùlarge_pool.freed‚1522131398896640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚342459955210240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11547729103360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647~62–'‰!ˆN XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98408549 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17359434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178907987 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178918146 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11530640201828 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1522131398896640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1524758561946906 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80499438 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 $ _timestamp‚1.784515848803579e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80499438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41109057 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23749626 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1522131398896640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚342459955210240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11547729103360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11530299640442 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1524758919547652 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178918146 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11612969774080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11548070534656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚342460079084544 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _runtime‚ 9022.37937293 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330847004048384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178907987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80499440 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11548070534656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330846985436160 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1513228262306464 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1522131415936000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98408549 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17359431 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41109346 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80499440 _step‚818 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98418706 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656²’ªÔ2Š!- _wandbruntime‚9022  _runtime‚9022é0È£ ‹!„ /loss/FRt1/assignment_loss‚1.615109926671838 2loss/SRqqt2/assignment_loss‚0.5228738667919198 1loss/SRqqt2/detection_loss‚0.5832815849258561 0loss/FBt2/detection_loss‚0.40845428603002804 _runtime‚9022.380120682 0loss/FRt2/assignment_loss‚0.7635039883565575 3loss/FBt2/assignment_loss‚0.0007936559249959884 1loss/SRqqt1/detection_loss‚0.7404640108128661 /loss/FBt1/detection_loss‚0.5055088528897613 % _timestamp‚1.7845158488047194e+09 0loss/FRt1/detection_loss‚0.46464795625610505 %loss/total_loss‚7.707574816974027 epoch‚24 trainer/global_step‚9949 1loss/SRqqt1/assignment_loss‚0.586586825751283 1loss/FBt1/assignment_loss‚0.01136931009011412 /loss/FRt2/detection_loss‚1.5049805524727027 _step‚819IŒ€wŒ!‰ 1loss/SRqqt2/detection_loss‚0.5832815849258561 0loss/FBt2/detection_loss‚0.40845428603002804 _runtime‚9022.380120682 0loss/FRt2/assignment_loss‚0.7635039883565575 3loss/FBt2/assignment_loss‚0.0007936559249959884 1loss/SRqqt1/detection_loss‚0.7404640108128661 /loss/FBt1/detection_loss‚0.5055088528897613 % _timestamp‚1.7845158488047194e+09 0loss/FRt1/detection_loss‚0.46464795625610505 %loss/total_loss‚7.707574816974027 epoch‚24 trainer/global_step‚9949 1loss/SRqqt1/assignment_loss‚0.586586825751283 1loss/FBt1/assignment_loss‚0.01136931009011412 /loss/FRt2/detection_loss‚1.5049805524727027 _step‚819 /loss/FRt1/assignment_loss‚1.615109926671838 2loss/SRqqt2/assignment_loss‚0.5228738667919198³Ûñü½!‚(:ô šöÒ݆‹Í" proc.memory.rssMB‚ 3022.3671875* proc.memory.percent‚18.447065353393555 cpu‚25.68542428916685 proc.cpu.threads‚17% disk./.usageGB‚1166.8622665405273 disk.nvme0n1.in‚ 134.98046875 disk.nvme1c1n1.in‚ 190.734375& disk.nvme1n1.out‚28202.72509765625 disk.md0.in‚0 disk.dm-0.in‚ 325.71484375 disk.dm-1.in‚0" disk.dm-1.out‚11781.6337890625 network.sent‚ 1.5820753e+07% disk.nvme0n1.out‚54053.3173828125( disk.nvme1c1n1.out‚28202.72509765625 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 134.98046875' disk.nvme0c0n1.out‚54053.3173828125( proc.memory.availableMB‚ 523.70703125 disk.md0.out‚ 505.93359375$ memory_percent‚96.80354595184326 network.recv‚ 2.389685e+06) disk./.usagePercent‚76.00470715131264 disk.nvme1n1.in‚ 190.734375# disk.dm-0.out‚69425.03076171875_³²$žŽ!‚(:“ šöÒåŸðÍ gpu.0.gpu‚43.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.854' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.28466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ü€œ‚!‚(:÷ ŸšöÒÿŒäË$ memory_percent‚96.80604934692383 proc.cpu.threads‚17 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 190.88671875( disk.nvme1c1n1.out‚28206.54541015625 disk.md0.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 3022.3671875 network.recv‚ 2.394259e+06) disk./.usagePercent‚76.00470640589062( disk.nvme0c0n1.out‚54054.41650390625 disk.md0.out‚ 506.2890625# disk.dm-1.out‚11784.95654296875& proc.memory.availableMB‚ 523.296875* proc.memory.percent‚18.447065353393555 network.sent‚ 1.5885057e+07 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 137.8203125 disk.nvme0n1.in‚ 137.8203125 disk.nvme1n1.in‚ 190.88671875# disk.dm-0.out‚69425.85888671875 cpu‚25.685042318247138% disk./.usageGB‚1166.8622550964355& disk.nvme0n1.out‚54054.41650390625& disk.nvme1n1.out‚28206.54541015625 disk.dm-0.in‚ 328.70703125kÝ!‚(:’ ŸšöÒóáâË gpu.0.gpu‚49.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.36' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.45333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸‹p‘!‚(:ö ®šöÒ×ÞÕË) disk./.usagePercent‚76.00473945293416" disk.dm-0.out‚69428.0966796875 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme1c1n1.out‚28207.3896484375 disk.nvme1n1.in‚ 200.41015625% disk.nvme1n1.out‚28207.3896484375" proc.memory.rssMB‚ 3022.3671875* proc.memory.percent‚18.447065353393555 cpu‚25.68465448902293% disk./.usageGB‚1166.8627624511719 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 137.94921875( disk.nvme0c0n1.out‚54056.71337890625" disk.nvme1c1n1.in‚ 200.41015625 disk.md0.in‚0 disk.md0.out‚ 506.66796875$ memory_percent‚96.80612087249756 network.sent‚ 1.5888021e+07 disk.dm-0.in‚ 338.359375# disk.dm-1.out‚11785.04638671875 disk.nvme0n1.in‚ 137.94921875& disk.nvme0n1.out‚54056.71337890625( proc.memory.availableMB‚ 523.28515625 proc.cpu.threads‚17 network.recv‚ 2.397463e+06Œ%ÕÌœ’!‚(:‘ ®šöÒ‰ø™Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.136' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.04533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Új]2“!- _wandbruntime‚9066  _runtime‚9066VV]€É”!à _step‚820 ' lr-AdamW/pg1‚0.00031714161001842754 ' lr-AdamW/pg2‚0.00031714161001842754 trainer/global_step‚9999 % _timestamp‚1.7845158926818209e+09 _runtime‚9066.257409472¹ò—€Î•!È ' lr-AdamW/pg1‚0.00031714161001842754 ' lr-AdamW/pg2‚0.00031714161001842754 trainer/global_step‚9999 % _timestamp‚1.7845158926818209e+09 _runtime‚9066.257409472 _step‚820´CÁ‚2–!- _wandbruntime‚9066  _runtime‚9066Ò½‚¯(—!ûO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98893688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11602529530368 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80881297 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98893688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11667643432960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1540999396071424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1540999037607936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80881295 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11584674465598 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17444400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚821 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98883545 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344140462103040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1529396866541056 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1529396849501696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11602188106240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179764840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98883545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344140590178816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80881295 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1532037039478190 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11585015026872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17444397 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚332472818670080 _runtime‚9066.258551634 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1540999396071424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179774985 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1520452365012592 trainer/global_step‚9999 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11602529530368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80881297 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41306869 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1532037397078824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1540999037607936ùdåÁâ˜!€P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80881295 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11584674465598 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17444400 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚821 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98883545 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344140462103040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1529396866541056 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1529396849501696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11602188106240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batò$ch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179764840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98883545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344140590178816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80881295 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1532037039478190 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11585015026872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17444397 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚332472818670080 _runtime‚9066.258551634 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1540999396071424 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179774985 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1520452365012592 trainer/global_step‚9999 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11602529530368 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80881297 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41306869 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1532037397078824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1540999037607936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98893688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11602529530368 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80881297 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98893688 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11667643432960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1540999396071424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1540999037607936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528µdïL92™!- _wandbruntime‚9066  _runtime‚9066èKqü 'š!ƒN U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1541148780679680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11668863953920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80889090 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 179782289 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344174757632000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1532185936134660 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80889088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1532185578533914 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41310635 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11668758691840 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1529545141043200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41310919 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17446131 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23864504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚332505998940160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98903358 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11603639636480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80889088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1529545124003840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179782289 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17446134 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11603298205184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1520599812719424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚9066.885308193 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1541148780679680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11603639636480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1529545124003840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11603298205184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11586123415236 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179792448 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1529545141043200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179792448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _step‚822 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9999 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80889090 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98893201 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1541148422209024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98903358 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11603639636480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80889088 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1529545124003840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179782289 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17446134 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11603298205184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1520599812719424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚9066.885308193 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1541148780679680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11603639636480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1529545124003840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11603298205184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11586123415236 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179792448 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1529545141043200 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179792448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _step‚822 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9999 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80889090 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98893201 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1541148422209024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1532185578533914 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41310635 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11668758691840 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1529545141043200 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41310919 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799¶øvçæ2œ!- _wandbruntime‚9066  _runtime‚9066)éƒI !‡ 0loss/FRt1/assignment_loss‚1.7016687727139401 0loss/FRt2/assignment_loss‚0.7141632959246635 1loss/SRqqt1/detection_loss‚0.7500535869027044 1loss/SRqqt2/detection_loss‚0.6408002835698425 /loss/FBt2/detection_loss‚0.8304671289487964 trainer/global_step‚9999 /loss/FRt2/detection_loss‚1.5184539932150594 0loss/FBt1/detection_loss‚0.31726908589371305 epoch‚24 _step‚823 3loss/FBt1/assignment_loss‚0.0040904512024168005 2loss/FBt2/assignment_loss‚0.003836941524169341 %loss/total_loss‚7.905900666913475 % _timestamp‚1.7845158933110425e+09 3loss/SRqqt1/assignment_loss‚0.45355006488049465 1loss/SRqqt2/assignment_loss‚0.522521126730756 0loss/FRt1/detection_loss‚0.44902593540691776 _runtime‚9066.886414405›Dž!Œ 3loss/SRqqt1/assignment_loss‚0.45355006488049465 1loss/SRqqt2/assignment_loss‚0.522521126730756 0loss/FRt1/detection_loss‚0.44902593540691776 _runtime‚9066.886414405 0loss/FRt1/assignment_loss‚1.7016687727139401 0loss/FRt2/assignment_loss‚0.7141632959246635 1loss/SRqqt1/detection_loss‚0.7500535869027044 1loss/SRqqt2/detection_loss‚0.6408002835698425 /loss/FBt2/detection_loss‚0.8304671289487964 trainer/global_step‚9999 /loss/FRt2/detection_loss‚1.5184539932150594 0loss/FBt1/detection_loss‚0.31726908589371305 epoch‚24 _step‚823 3loss/FBt1/assignment_loss‚0.0040904512024168005 2loss/FBt2/assignment_loss‚0.003836941524169341 %loss/total_loss‚7.905900666913475 % _timestamp‚1.7845158933110425e+09·E˜FŸ!‚(:ö ½šöÒáÄ€Ì" disk.nvme1c1n1.in‚ 200.41015625( disk.nvme1c1n1.out‚28208.12255859375 network.sent‚ 1.5905701e+07 disk.md0.out‚ 507.0078125 disk.dm-0.in‚ 358.015625" disk.nvme0c0n1.in‚ 157.60546875( disk.nvme0c0n1.out‚54057.55810546875 disk.nvme0n1.in‚ 157.60546875$ memory_percent‚96.80485725402832) disk./.usagePercent‚76.00468329780752% disk./.usageGB‚1166.8619003295898" disk.dm-1.out‚11785.2861328125* proc.memory.percent‚18.447065353393555 cpu‚25.686520252038584 proc.cpu.threads‚17 network.recv‚ 2.401331e+06 disk.nvme1n1.in‚ 200.41015625 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚54057.55810546875' proc.memory.availableMB‚ 523.4921875" proc.memory.rssMB‚ 3022.3671875& disk.nvme1n1.out‚28208.12255859375 disk.md0.in‚0" disk.dm-0.out‚69428.6826171875 disk.rbd0.out‚0-Y1› !‚(: ½šöÒëö¿Ì gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.03' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.67666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å¶ü2¡!- _wandbruntime‚9083  _runtime‚9083ªmá §¢!ûz 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6884530782699585 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48651736974716187 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2664421796798706 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 epoch‚24 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.29789507389068604 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.915117084980011 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25625500082969666 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.9050453901290894 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.778272807598114 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_2_of_2‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8672560453414917 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚9999 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7315012812614441 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7452110052108765 &particle/f_score‚0.661702036857605 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2544011175632477 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.9023053050041199 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3100605607032776 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26794669032096863 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8912131190299988 particle/accuracy_2_of_5‚1 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458730697632 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9559962749481201 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25444263219833374 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2515202760696411 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03558842092752457 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7419310212135315 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.7095803618431091 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5488660335540771 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963907599449158 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚9083.394809117 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2890174984931946 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03770478814840317 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9914441704750061 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7422742247581482 *jet/accuracy_2_of_4‚0.7807855010032654 (particle/accuracy‚0.8564839959144592 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04592081904411316 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBtf²²Ú(/SRqqt_purity‚0.7920533418655396 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09303852170705795 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +particle/sensitivity‚0.6536523103713989 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24405445158481598 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2979448437690735 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5558332204818726 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3290215730667114 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30877622961997986 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.055469825863838196 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17781220376491547 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6938613653182983 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03767116740345955 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5226229429244995 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7265788912773132 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_3‚0.5845028758049011 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984579682350159 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6804920434951782 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 5 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.59904944896698 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9348879456520081 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7272753715515137 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31411823630332947 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6567551493644714 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10844041407108307 /particle/accuracy_3_of_4‚0.9974937438964844 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7665177583694458 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977699518203735 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2979448437690735 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9110544323921204 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7056410908699036 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7565166354179382 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.058311302214860916 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16567383706569672 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 particle/accuracy_2_of_4‚1 6validation_average_jet_accuracy‚0.3984415531158447 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8472390174865723 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8930563926696777 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5329545140266418 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7921975255012512 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29607677459716797 /particle/accuracy_4_of_4‚0.9569149017333984 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9955584406852722 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5887559652328491 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.394731342792511 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03787633776664734 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.731033980846405 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.768212080001831 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9071065783500671 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6881561279296875 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962657690048218 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.671100378036499 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22605310380458832 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6897249221801758 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4648769497871399 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30959269404411316 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7221253514289856 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1803797334432602 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9896988868713379 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.047658152878284454 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7391681671142578 +jet/accuracy_1_of_1‚0.35883909463882446 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2394704818725586 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 _step‚824 )loss/val_total_loss‚7.660803968675618 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3920426070690155 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8472390174865723 +jet/accuracy_3_of_4‚0.09332889318466187 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19255563616752625 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.646080732345581 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7401421070098877 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8124716877937317 *jet/accuracy_2_of_2‚0.1699400246143341 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3527986407279968 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7596908807754517 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9897502660751343 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.8122352361679077 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23622308671474457 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 +jet/accuracy_3_of_3‚0.07885583490133286 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7049214243888855 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48616617918014526 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6576251983642578 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7835099101066589 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +particle/specificity‚0.9119530916213989 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7436380982398987 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6599741578102112 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6710204482078552 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941349029541016 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23944391310214996 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2650364339351654 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03558842092752457 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 *jet/accuracy_1_of_4‚0.9871495366096497 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6881238222122192 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5702280402183533 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3379547595977783 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9931564331054688 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6512660384178162 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.36320582032203674 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03742515668272972 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6576251983642578 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 % _timestamp‚1.7845159098186612e+09 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_5_of_5‚1Y0z†=£!€{ 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17781220376491547 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6938613653182983 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03767116740345955 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5226229429244995 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7265788912773132 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_3‚0.5845028758049011 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984579682350159 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6804920434951782 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 5 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.59904944896698 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9348879456520081 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7272753715515137 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31411823630332947 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6567551493644714 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10844041407108307 /particle/accuracy_3_of_4‚0.9974937438964844 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7665177583694458 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9977699518203735 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2979448437690735 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.9110544323921204 particle/accuracy_1_of_1‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7056410908699036 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7565166354179382 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.058311302214860916 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16567383706569672 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 particle/accuracy_2_of_4‚1 6validation_average_jet_accuracy‚0.3984415531158447 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8472390174865723 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8930563926696777 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5329545140266418 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7921975255012512 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29607677459716797 /particle/accuracy_4_of_4‚0.9569149017333984 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9955584406852722 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5887559652328491 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.394731342792511 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03787633776664734 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.731033980846405 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.768212080001831 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9071065783500671 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6881561279296875 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962657690048218 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.671100378036499 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22605310380458832 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6897249221801758 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4648769497871399 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30959269404411316 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7221253514289856 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1803797334432602 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9896988868713379 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.047658152878284454 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7391681671142578 +jet/accuracy_1_of_1‚0.35883909463882446 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2394704818725586 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 _step‚824 )loss/val_total_loss‚7.660803968675618 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3920426070690155 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8472390174865723 +jet/accuracy_3_of_4‚0.09332889318466187 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19255563616752625 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.646080732345581 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7401421070098877 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8124716877937317 *jet/accuracy_2_of_2‚0.1699400246143341 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3527986407279968 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7596908807754517 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9897502660751343 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.8122352361679077 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23622308671474457 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 +jet/accuracy_3_of_3‚0.07885583490133286 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7049214243888855 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48616617918014526 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6576251983642578 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7835099101066589 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +particle/specificity‚0.9119530916213989 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7436380982398987 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6599741578102112 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6710204482078552 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941349029541016 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23944391310214996 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2650364339351654 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03558842092752457 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 *jet/accuracy_1_of_4‚0.9871495366096497 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6881238222122192 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5702280402183533 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3379547595977783 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9931564331054688 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6512660384178162 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.36320582032203674 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03742515668272972 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6576251983642578 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 % _timestamp‚1.7845159098186612e+09 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_5_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6884530782699585 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48651736974716187 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2664421796798706 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 epoch‚24 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.29789507389068604 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.915117084980011 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25625500082969666 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.9050453901290894 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.778272807598114 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_2_of_2‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8672560453414917 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚9999 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7315012812614441 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7452110052108765 &particle/f_score‚0.661702036857605 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2544011175632477 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.9023053050041199 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3100605607032776 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26794669032096863 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8912131190299988 particle/accuracy_2_of_5‚1 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458730697632 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9559962749481201 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.25444263219833374 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2515202760696411 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03558842092752457 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7419310212135315 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.7095803618431091 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5488660335540771 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963907599449158 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40509259700775146 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚9083.394809117 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2890174984931946 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03770478814840317 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9914441704750061 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7422742247581482 *jet/accuracy_2_of_4‚0.7807855010032654 (particle/accuracy‚0.8564839959144592 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04592081904411316 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7920533418655396 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09303852170705795 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +particle/sensitivity‚0.6536523103713989 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.24405445158481598 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2979448437690735 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5558332204818726 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3290215730667114 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30877622961997986 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.055469825863838196 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1¸Ñš…~¤!‚(:ó ÌšöÒÆ°ÛË% disk.nvme0n1.out‚54059.8212890625 disk.nvme1n1.in‚ 217.30859375) disk./.usagePercent‚76.00465472329616% disk./.usageGB‚1166.8614616394043# disk.dm-0.out‚69430.78759765625' disk.nvme0c0n1.out‚54059.8212890625( disk.nvme1c1n1.out‚28454.22216796875 disk.md0.out‚ 507.40234375 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 217.30859375& disk.nvme1n1.out‚28454.22216796875$ memory_percent‚95.48444747924805& proc.memory.availableMB‚ 739.828125 proc.memory.rssMB‚ 3023.6875 proc.cpu.threads‚17 network.sent‚ 1.5968161e+07 network.recv‚ 2.405631e+06 disk.dm-1.in‚0" disk.dm-1.out‚12030.6962890625 disk.md0.in‚0* proc.memory.percent‚18.455123901367188 cpu‚25.688246306364594 disk.dm-0.in‚ 374.9140625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 157.60546875 disk.nvme0n1.in‚ 157.60546875’òÉ[ž¥!‚(:“ ÌšöÒ߈ Ì gpu.0.gpu‚91.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚279.351' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.11699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SŠÛ{¦!‚(:ð ÛšöÒñ…ÔË# disk.dm-1.out‚12030.78466796875 disk.rbd0.out‚0 network.recv‚ 2.409996e+06 disk.nvme0n1.in‚ 157.60546875$ disk.nvme0n1.out‚54060.662109375" disk.nvme0c0n1.in‚ 157.60546875& disk.nvme0c0n1.out‚54060.662109375( proc.memory.availableMB‚ 739.83203125 cpu‚25.68788224796352# disk.dm-0.out‚69431.43212890625 network.sent‚ 1.601442e+07% disk./.usageGB‚1166.6239433288574' disk.nvme1c1n1.out‚28454.7255859375 disk.nvme1n1.in‚ 217.30859375 disk.md0.in‚0 disk.dm-0.in‚ 374.9140625$ memory_percent‚95.48442363739014% disk.nvme1n1.out‚28454.7255859375 disk.md0.out‚ 507.68359375 disk.rbd0.in‚0 proc.memory.rssMB‚ 3023.6875* proc.memory.percent‚18.455123901367188 proc.cpu.threads‚17) disk./.usagePercent‚75.98918373742761" disk.nvme1c1n1.in‚ 217.30859375 disk.dm-1.in‚0+ !Ùž§!‚(:“ ÛšöÒ­…Ì gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚287.086' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.69533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒÕó{¨!‚(:ð êšöÒ‘®¦Ë disk.dm-1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚18.455123901367188$ disk./.usageGB‚1166.623950958252' disk.nvme1c1n1.out‚28700.7275390625% disk.nvme1n1.out‚28700.7275390625 disk.md0.out‚ 507.984375 disk.dm-0.in‚ 374.9140625# disk.dm-0.out‚69435.22412109375# disk.dm-1.out‚12276.22607421875" disk.nvme0c0n1.in‚ 157.60546875 network.sent‚ 1.6017384e+07) disk./.usagePercent‚75.98918423437564 disk.md0.in‚0' disk.nvme0c0n1.out‚54064.5458984375 disk.nvme0n1.in‚ 157.60546875% disk.nvme0n1.out‚54064.5458984375( proc.memory.availableMB‚ 739.83203125 proc.memory.rssMB‚ 3023.6875 network.recv‚ 2.41302e+06" disk.nvme1c1n1.in‚ 217.30859375 disk.nvme1n1.in‚ 217.30859375 disk.rbd0.out‚0$ memory_percent‚95.48442363739014 cpu‚25.687493800701304 proc.cpu.threads‚17ÛQ¿–“©!‚(:ˆ êšöÒȳöË gpu.0.gpu‚95.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚274.713' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.571 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þS[Ÿ2ª!- _wandbruntime‚9128  _runtime‚9128~ Ü*Ê«!Ä ' lr-AdamW/pg1‚0.00031468025518672105 ' lr-AdamW/pg2‚0.00031468025518672105 trainer/global_step‚10049 % _timestamp‚1.7845159547647908e+09 _runtime‚9128.340418587 _step‚825Ë@,ãϬ!É % _timestamp‚1.7845159547647908e+09 _runtime‚9128.340418587 _step‚825 ' lr-AdamW/pg1‚0.00031468025518672105 ' lr-AdamW/pg2‚0.00031468025518672105 trainer/global_step‚10049¹’1…2­!- _wandbruntime‚9128  _runtime‚9128qGy®!ûO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99449696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345898663855104 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚826 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1529745171433776 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1529745154394416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41547132 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180853594 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24011919 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345898777250816 KBDeviceSt* ;üatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1538752935523840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334154429448704 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1541405615922324 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11660461527908 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11678416470528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11744347802112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99459839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1538752918484480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17535216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41547423 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81393753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11660802089182 trainer/global_step‚10049 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11678416470528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1541405973522958 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 % _timestamp‚1.7845159547657294e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1550431351994368 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1538752918484480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11744253018624 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99459839 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1550430993530880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24012207 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1550430993530880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81393755 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143ÐÇhr(¯!€P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24012207 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1550430993530880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81393755 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99449696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345898663855104 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚826 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1529745171433776 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1529745154394416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41547132 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180853594 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24011919 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345898777250816 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1538752935523840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚334154429448704 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1541405615922324 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11660461527908 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11678416470528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11744347802112 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99459839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1538752918484480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17535216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41547423 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81393753 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11660802089182 trainer/global_step‚10049 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11678416470528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1541405973522958 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 % _timestamp‚1.7845159547657294e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1550431351994368 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1538752918484480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11744253018624 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99459839 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1550430993530880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496ºúºÂc2°!- _wandbruntime‚9128  _runtime‚9128á%a% '±!…N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1550580736602624 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81401546 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180871057 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17536950 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚10049 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1538901192986624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345932962568704 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180860898 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345933078054400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11679526576640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180871057 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1550580736602624 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1538901210025984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99459352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99469509 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81401548 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180860898 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1541554512578794 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11745371462144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11661569916160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1538901210025984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17536947 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24014283 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11679185145344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1550580378131968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11745468335616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41551230 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0Pg¸5ž²!ŠN J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81401546 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11679185145344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99469509 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1541554154978048 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180860898 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345933078054400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11679526576640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180871057 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1550580736602624 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1538901210025984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99459352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99469509 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81401548 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180860898 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1541554512578794 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11745371462144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11661569916160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1538901210025984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17536947 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24014283 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11679185145344 T?DevÞÓìßriceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1550580378131968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11745468335616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41551230 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1550580736602624 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81401546 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180871057 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17536950 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚10049 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1538901192986624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345932962568704 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1557900287739904 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11733644148224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1557900646203392 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81783403 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24127685 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99934348 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11800146147840 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11716285302590 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1546166660631040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1546166643591680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81783403 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚335813442952704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1537116704807376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24127420 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚10099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181727896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99934348 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1557900287739904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347613570488320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚9172.319065214 _step‚830 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11733985572352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17621913 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1557900646203392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚335813424340480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 % _timestamp‚1.7845159987398264e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17621916 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99944491 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81783405 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81783405 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41749333 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1546166660631040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99934348 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1557900287739904 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347613570488320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚9172.319065214 _step‚830 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11733985572352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17621913 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1557900646203392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚335813424340480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 % _timestamp‚1.7845159987398264e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17621916 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99944491 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81783405 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81783405 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41749333 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1546166660631040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1557900287739904 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11733644148224 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1557900646203392 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81783403 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24127685 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99934348 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11800146147840 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11716285302590 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0¾,ÁJœ2Â!- _wandbruntime‚9172  _runtime‚9172Æ)¸}QÃ!†N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 E‹ñõz»%=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41753600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11801251428864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1546314918093824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 % _timestamp‚1.7845159993630538e+09 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181745359 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99954161 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347647984108032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99944004 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚335846604610560 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1546314935133184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1548981171565056 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181735200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81791196 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99954161 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1537264118435488 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17623647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11717053129568 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1558050030811648 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1558050030811648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99944004 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11734754247168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81791196 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1546314935133184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41753321 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11801360885248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1546314918093824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 % _timestamp‚1.7845159993630538e+09 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181745359 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99954161 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚347647984108032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99944004 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚335846604610560 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1546314935133184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1548981171565056 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181735200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81791196 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99954161 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1537264118435488 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17623647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11717053129568 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1558050030811648 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1558050030811648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99944004 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11734754247168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81791196 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1546314935133184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41753321 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11801360885248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1565369581948928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24243087 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11856071678976 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11771768515998 _step‚834 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11855964312576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚349328402157056 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚337472456456704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚9215.458644007 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82173053 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17708613 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100419000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 % _timestamp‚1.7845160418831797e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11789213250048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17708616 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1544488221141616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11789213250048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10149 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1553580368698880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚337472437844480 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1556259649096340 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182602198 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11771427954724 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11789554674176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41951413 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1565369940412416 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182592053 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100419000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82173055 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚349328528135680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1565369940412416 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1565369581948928 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1544488238180976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1553580385738240 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82173053 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82173055x~÷F (Ó!…P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1565369581948928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24243087 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11856071678976 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11771768515998 _step‚834 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11855964312576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚349328402157056 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚337472456456704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚9215.458644007 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82173053 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17708613 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100419000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 % _timestamp‚1.7845160418831797e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11789213250048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17708616 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1544488221141616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11789213250048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10149 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1553580368698880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚337472437844480 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1556259649096340 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182602198 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11771427954724 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11789554674176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41951413 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1565369940412416 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182592053 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100419000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82173055 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚349328528135680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1565369940412416 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1565369581948928 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1544488238180976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1553580385738240 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82173053 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82173055 mC2Ô!- _wandbruntime‚9216  _runtime‚9216=óˆ'Õ!‰N [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1544635651809088 trainer/global_step‚10149 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100438813 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182609502 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚337505636726784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41955750 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1553728643201024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 $ _timestamp‚1.784516042492704e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚349362822952960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82180846 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1553728643201024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1565519325020672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100428656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _step‚835 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11857074672640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11790664780288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11790323348992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100438813 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82180848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17710350 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11772876904362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24245400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11790664780288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82180846 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349362692787200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182609502 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182619661 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1544635668848448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1565518966550016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17710347 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11772536342976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11790323348992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41955467 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182619661 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚9216.068485427 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1556408188152064 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1553728660240384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1556408545752810 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1565519325020672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚337505618114560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚337505636726784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41955750 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1553728643201024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 $ _timestamp‚1.784516042492704e+09 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚349362822952960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82180846 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1553728643201024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1565519325020672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100428656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _step‚835 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11857074672640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11790664780288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11790323348992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100438813 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82180848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17710350 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11772876904362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24245400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11790664780288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82180846 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349362692787200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182609502 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182619661 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1544635668848448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1565518966550016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17710347 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11772536342976 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11790323348992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41955467 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182619661 QDevdø¢®?iceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚9216.068485427 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1556408188152064 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1553728660240384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1556408545752810 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1565519325020672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ÃÔ‡ 2×!- _wandbruntime‚9216  _runtime‚9216…òkŸØ!‚ epoch‚25 0loss/FRt2/assignment_loss‚0.8718785539982601 $loss/total_loss‚8.31043103630827 trainer/global_step‚10149 _runtime‚9216.069360428 _step‚836 0loss/FRt1/assignment_loss‚1.5465902277067596 2loss/SRqqt1/assignment_loss‚0.5573954520554187 1loss/SRqqt2/detection_loss‚0.7484359634844205 /loss/FBt1/detection_loss‚0.5690087115169381 /loss/FBt2/detection_loss‚0.6688219287898391 % _timestamp‚1.7845160424940746e+09 2loss/SRqqt2/assignment_loss‚0.5331005073317057 1loss/FBt1/assignment_loss‚0.07307097560132769 .loss/FRt1/detection_loss‚0.471680416311597 1loss/SRqqt1/detection_loss‚0.6553299736931096 2loss/FBt2/assignment_loss‚0.013644638399334497 /loss/FRt2/detection_loss‚1.6014736874195599b7Š Ù!‡ 1loss/SRqqt1/detection_loss‚0.6553299736931096 2loss/FBt2/assignment_loss‚0.013644638399334497 /loss/FRt2/detection_loss‚1.6014736874195599 epoch‚25 0loss/FRt2/assignment_loss‚0.8718785539982601 $loss/total_loss‚8.31043103630827 trainer/global_step‚10149 _runtime‚9216.069360428 _step‚836 0loss/FRt1/assignment_loss‚1.5465902277067596 2loss/SRqqt1/assignment_loss‚0.5573954520554187 1loss/SRqqt2/detection_loss‚0.7484359634844205 /loss/FBt1/detection_loss‚0.5690087115169381 /loss/FBt2/detection_loss‚0.6688219287898391 % _timestamp‚1.7845160424940746e+09 2loss/SRqqt2/assignment_loss‚0.5331005073317057 1loss/FBt1/assignment_loss‚0.07307097560132769 .loss/FRt1/detection_loss‚0.471680416311597ÄXA YsÚ!‚(:è Ó›öÒ‘ÝÜË disk.dm-0.in‚ 375.17578125# disk.dm-0.out‚70740.66259765625 disk.nvme0n1.in‚ 157.8671875 disk.nvme1n1.in‚ 217.30859375$ memory_percent‚95.49098014831543 proc.cpu.threads‚17 disk.dm-1.out‚12345.26953125$ disk.nvme0c0n1.out‚ 54259.7734375" disk.nvme0n1.out‚ 54259.7734375" disk.nvme1c1n1.in‚ 217.30859375( disk.nvme1c1n1.out‚29895.30224609375& disk.nvme1n1.out‚29895.30224609375 disk.md0.out‚ 514.671875 proc.memory.rssMB‚ 3023.6875 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 157.8671875 disk.md0.in‚0* proc.memory.percent‚18.455123901367188 network.recv‚ 2.439534e+06) disk./.usagePercent‚76.09251662470041 disk.dm-1.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 738.7578125 cpu‚25.689635318229996 network.sent‚ 1.6180565e+07$ disk./.usageGB‚1168.210361480713ögžÛ!‚(:“ Ó›öÒÍõÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚189.458' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.15266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«¾ýtÜ!‚(:é â›öÒ­ÅË& disk.nvme0c0n1.out‚54268.552734375 disk.nvme0n1.in‚ 162.0859375$ disk.nvme0n1.out‚54268.552734375" disk.nvme1c1n1.in‚ 217.30859375 disk.md0.out‚ 515.3046875' disk.nvme1c1n1.out‚30247.5205078125" disk.dm-0.out‚71090.9111328125 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚96.99294567108154( proc.memory.availableMB‚ 492.67578125 proc.memory.rssMB‚ 3023.6875% disk.nvme1n1.out‚30247.5205078125 disk.dm-0.in‚ 379.39453125 disk.rbd0.in‚0* proc.memory.percent‚18.455123901367188 network.sent‚ 1.6230826e+07$ disk./.usageGB‚1168.198398590088 disk.nvme1n1.in‚ 217.30859375 disk.dm-1.out‚12356.18359375 cpu‚25.68962617676048 proc.cpu.threads‚17 disk.md0.in‚0 network.recv‚ 2.442851e+06( disk./.usagePercent‚76.0917374101995! disk.nvme0c0n1.in‚ 162.0859375öh ZÝ!‚(:’ â›öÒÇÚ…Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚166.109' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚55.369666666666674 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à†L—zÞ!‚(:ï ñ›öÒ•é“Ë proc.memory.rssMB‚ 3023.6875* proc.memory.percent‚18.455123901367188 disk.md0.in‚0 disk.md0.out‚ 515.71875 disk.dm-1.in‚0( disk.nvme0c0n1.out‚54277.18505859375 network.sent‚ 1.6233002e+07) disk./.usagePercent‚76.09167752796266% disk./.usageGB‚1168.1974792480469 disk.nvme1n1.in‚ 217.30859375$ disk.nvme1n1.out‚30248.802734375 disk.dm-0.in‚ 379.39453125# disk.dm-1.out‚12356.89990234375 disk.rbd0.out‚0 network.recv‚ 2.446081e+06 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 162.0859375 disk.nvme0n1.in‚ 162.0859375& disk.nvme0n1.out‚54277.18505859375" disk.nvme1c1n1.in‚ 217.30859375 cpu‚25.689753063482847 proc.cpu.threads‚17" disk.dm-0.out‚71099.1962890625& disk.nvme1c1n1.out‚30248.802734375$ memory_percent‚96.99246883392334( proc.memory.availableMB‚ 492.75390625ª™1žß!‚(:“ ñ›öÒÙ˜éË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚287.669' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.88966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06‰é2à!- _wandbruntime‚9258  _runtime‚9258K݆üÈá! _runtime‚9258.605592751 _step‚837 & lr-AdamW/pg1‚0.0003072939577610232 & lr-AdamW/pg2‚0.0003072939577610232 trainer/global_step‚10199 % _timestamp‚1.7845160850299754e+09ùܤ$Íâ!Ç & lr-AdamW/pg1‚0.0003072939577610232 & lr-AdamW/pg2‚0.0003072939577610232 trainer/global_step‚10199 % _timestamp‚1.7845160850299754e+09 _runtime‚9258.605592751 _step‚837ÅBi£2ã!- _wandbruntime‚9258  _runtime‚9258Þ%Ba(ä!þO aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1560994110845440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100913795 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82562705 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82562703 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11845123776000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚351043204258816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1572838876157952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100913795 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82562705 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11911858179584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚838 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1560994093806080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183476500 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100903652 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339131469960704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1560994093806080 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1572839234621440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42153043 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚351043328140288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7845160850309687e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1572839234621440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11911752910336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42152753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11844782351872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11827251729406 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1572838876157952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17795316 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100903652 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1563687023283982 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1560994110845440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11845123776000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339131451348480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17795313 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1551859754515216 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1551859771554576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82562703 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11844782351872 trainer/global_step‚10199‚1þö (å!ƒP QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100913795 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82562705 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11911858179584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚838 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1560994093806080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183476500 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100903652 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339131469960704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1560994093806080 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1572839234621440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42153043 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚351043328140288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7845160850309687e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1572839234621440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11911752910336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42152753 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11844782351872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11827251729406 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1572838876157952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17795316 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100903652 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1563687023283982 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1560994110845440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11845123776000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339131451348480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17795313 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1551859754515216 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1551859771554576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82562703 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11844782351872 trainer/global_step‚10199 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1560994110845440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100913795 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82562705 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82562703 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11845123776000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚351043204258816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1572838876157952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452ÆûÛn2æ!- _wandbruntime‚9259  _runtime‚9259%ÜñΊç!‰N ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11828360117770 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100913308 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1561142368308224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183483804 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17797050 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11912869332480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1572988619229696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42156757 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24359710 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100923465 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1561142368308224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100923465 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100913308 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24359987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11912972497408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183493963 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82570496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42157037 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚351077622728192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 % _timestamp‚1.7845160856405466e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11846233882112 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1563835204739072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocaõPÙ…ted‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1561142385347584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183483804 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11846233882112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11845892450816 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚10199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339164650230784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339164631618560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17797047 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183493963 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82570496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1552007202222048 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1552007185182688 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42156757 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24359710 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100923465 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1561142368308224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100923465 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100913308 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24359987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11912972497408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183493963 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82570496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42157037 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚351077622728192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 % _timestamp‚1.7845160856405466e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11846233882112 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1563835204739072 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1561142385347584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183483804 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11846233882112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11845892450816 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚10199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339164650230784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339164631618560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17797047 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183493963 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82570496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1552007202222048 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1552007185182688 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100913308 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1561142368308224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183483804 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17797050 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11912869332480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1572988619229696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42354060 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11882734942814 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1568407835952640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101388304 _step‚842 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚340790483464704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚340790464852480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _runtime‚9301.627109103 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24472333 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352758056735232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1568407818913280 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1568407835952640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82952355 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24472047 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11967690860544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82952353 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1580308528830464 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1580308528830464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82952355 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11967591882752 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1580308170366976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17882016 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1580308170366976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11900692877824 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11900692877824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1571114039870990 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17882013 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184350802 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11900351453696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42354349 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289Dö¶l÷!„P VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184350802 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11900351453696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42354349 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1559231304928176 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11882394381540 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7845161280505488e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XEDeviceStatsMonitor.on_train_batch_start/active_byte9¼8ž!s.small_pool.freed‚11900351453696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184340657 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352758174325248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42354060 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11882734942814 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1568407835952640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101388304 _step‚842 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚340790483464704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚340790464852480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _runtime‚9301.627109103 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24472333 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352758056735232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1568407818913280 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1568407835952640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82952355 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24472047 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11967690860544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82952353 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1580308528830464 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1580308528830464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82952355 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11967591882752 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1580308170366976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17882016 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1580308170366976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11900692877824 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11900692877824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1571114039870990 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17882013Ê⛯2ø!- _wandbruntime‚9302  _runtime‚9302ÐL ´'ù!‰N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚9302.228742194 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184358106 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184358106 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1568556093415424 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82960146 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 trainer/global_step‚10249 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11901802983936 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1580457554968064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1559378735595648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17883750 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352792470486016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340823663734784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24474578 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340823645122560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _step‚843 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1559378718556288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17883747 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184368265 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101408117 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352792352903168 % _timestamp‚1.7845161286529741e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1580457913438720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24474295 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11883502769792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101397960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11883843331178 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184368265 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11901461552640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1568556110454784 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1568556093415424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42358042Þ`x:'ú!ŽN VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17883750 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352792470486016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340823663734784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24474578 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340823645122560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _step‚843 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1559378718556288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17883747 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184368265 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101408117 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352792352903168 % _timestamp‚1.7845161286529741e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1580457913438720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24474295 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11883502769792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101397960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11883843331178 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184368265 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11901461552640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1568556110454784 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1568556093415424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42358042 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚9302.228742194 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184358106 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184358106 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1568556093415424 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82960146 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 trainer/global_step‚10249 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11901802983936 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1580457554968064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1559378735595648Ë–ì6Ë2û!- _wandbruntime‚9302  _runtime‚9302²‹é_ ü!„ % _timestamp‚1.7845161286619127e+09 _step‚844 2loss/SRqqt2/assignment_loss‚0.5222947574426723 .loss/FBt2/detection_loss‚0.478058859705925 _runtime‚9302.237303254 0loss/FRt1/assignment_loss‚1.6343805303760603 3loss/SRqqt1/assignment_loss‚0.47452653446095533 1loss/FBt1/assignment_loss‚0.02270259885460781 /loss/FRt2/detection_loss‚1.4855666206185432 0loss/FRt2/assignment_loss‚0.8000150122362024 /loss/FRt1/detection_loss‚0.4860369443218837 epoch‚25 3loss/FBt2/assignment_loss‚0.0033321083570863243 1loss/SRqqt1/detection_loss‚0.7237509996814772 1loss/SRqqt2/detection_loss‚0.6734874469862768 .loss/FBt1/detection_loss‚0.482986345132393 %loss/total_loss‚7.787138758174084 trainer/global_step‚10249|«(ðý!‰ 3loss/FBt2/assignment_loss‚0.0033321083570863243 1loss/SRqqt1/detection_loss‚0.7237509996814772 1loss/SRqqt2/detection_loss‚0.6734874469862768 .loss/FBt1/detection_loss‚0.482986345132393 %loss/total_loss‚7.787138758174084 trainer/global_step‚10249 % _timestamp‚1.7845161286619127e+09 _step‚844 2loss/SRqqt2/assignment_loss‚0.5222947574426723 .loss/FBt2/detection_loss‚0.478058859705925 _runtime‚9302.237303254 0loss/FRt1/assignment_loss‚1.6343805303760603 3loss/SRqqt1/assignment_loss‚0.47452653446095533 1loss/FBt1/assignment_loss‚0.02270259885460781 /loss/FRt2/detection_loss‚1.4855666206185432 0loss/FRt2/assignment_loss‚0.8000150122362024 /loss/FRt1/detection_loss‚0.4860369443218837 epoch‚25ÌHæ~=yþ!‚(:î ­œöÒü˜âË disk.nvme0n1.in‚ 162.0859375& disk.nvme1n1.out‚30271.54443359375 proc.cpu.threads‚17) disk./.usagePercent‚75.98937307462458% disk./.usageGB‚1166.6268501281738 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 162.0859375 disk.nvme1n1.in‚ 217.30859375$ memory_percent‚96.98994159698486 proc.memory.rssMB‚ 3023.6875 network.sent‚ 1.6322994e+07 disk.dm-0.in‚ 379.39453125# disk.dm-0.out‚71110.87158203125! disk.dm-1.out‚12372.904296875" disk.nvme1c1n1.in‚ 217.30859375( disk.nvme1c1n1.out‚30271.54443359375 cpu‚25.69225346800542# disk.nvme0n1.out‚54291.79296875 disk.md0.in‚0 disk.md0.out‚ 520.42578125( proc.memory.availableMB‚ 493.16796875* proc.memory.percent‚18.455123901367188 network.recv‚ 2.460776e+06 disk.dm-1.in‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚54291.79296875UÂÛžÿ!‚(:“ ­œöÒºó£Ì gpu.0.gpu‚83.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.824' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.27466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓYýy€"‚(:î ¼œöÒ›¿ÆË! disk.dm-1.out‚12372.951171875* proc.memory.percent‚18.455123901367188 cpu‚25.69306961674137 disk.rbd0.out‚0' disk.nvme0c0n1.out‚54296.4873046875( disk.nvme1c1n1.out‚30272.04833984375 disk.md0.out‚ 520.640625% disk.nvme0n1.out‚54296.4873046875 network.sent‚ 1.6372589e+07) disk./.usagePercent‚75.97389364063963 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme1n1.out‚30272.04833984375 disk.md0.in‚0( proc.memory.availableMB‚ 493.16796875 network.recv‚ 2.464849e+06 disk.nvme0n1.in‚ 162.0859375" disk.nvme1c1n1.in‚ 217.30859375 disk.nvme1n1.in‚ 217.30859375$ memory_percent‚96.98994159698486 proc.memory.rssMB‚ 3023.6875 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 162.0859375# disk./.usageGB‚1166.38920211792 disk.dm-0.in‚ 379.39453125# disk.dm-0.out‚71115.53857421875'Ré¼å"‚(:“ ¼œöÒÙ•Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚145.33' gpu.0.enforcedPower¢™•U¹LimitWatts‚300.0) gpu.0.powerPercent‚48.443333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aHsº2‚"- _wandbruntime‚9343  _runtime‚9343¦òïuȃ" & lr-AdamW/pg2‚0.0003023694800862221 trainer/global_step‚10299 % _timestamp‚1.7845161697241347e+09 _runtime‚9343.299647041 _step‚845 & lr-AdamW/pg1‚0.0003023694800862221]­™šÍ„"Ç & lr-AdamW/pg1‚0.0003023694800862221 & lr-AdamW/pg2‚0.0003023694800862221 trainer/global_step‚10299 % _timestamp‚1.7845161697241347e+09 _runtime‚9343.299647041 _step‚845Íš‰ 2…"- _wandbruntime‚9343  _runtime‚9343µ„ (†"€P _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11938218156222 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42556104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354472914582016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101872956 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24587391 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11955920555520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42556391 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342449478356480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12023436225536 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1578541056457998 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1575821544020480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17968716 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1587777823039488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11956261979648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83342005 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1566602838301776 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚10299 % _timestamp‚1.7845161697247033e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1587777464576000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1566602821262416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185225104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360—í\… (‡"…P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185225104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11938218156222 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42556104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354472914582016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101872956 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24587391 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11955920555520 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42556391 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342449478356480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12023436225536 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1578541056457998 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1575821544020480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17968716 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1587777823039488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11956261979648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83342005 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1566602838301776 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚10299 % _timestamp‚1.7845161697247033e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1587777464576000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1566602821262416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528Î}UÞ2ˆ"- _wandbruntime‚9343  _runtime‚9343h_üþ'‰"‰N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185242567 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185242567 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11957030654464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1578689595513834 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11938985983200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185232408 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101882612 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17970447 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42560072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24589907 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1566750268969248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11939326544586 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1566750251929888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1575969835561984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17970450 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101892769 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354507205212160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83349798 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.784516170324518e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1575969818522624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42560357 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1587927207647744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185232408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚9343.900213166 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1575969835561984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11957372085760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚847 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342482677238784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚10299 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1587926849177088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1578689237913088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83349796 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101892769 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180Þ*îN¶Š"ŽN L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1587927207647744 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185232408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚9343.900213166 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1575969835561984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11957372085760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚847 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342482677238784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚10299 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1587926849177088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1578689237913088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83349796 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101892769 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185242567 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185242567 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11957030654464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1578689595513834 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11938985983200 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185232408 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101882612 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17970447 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42560072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24589907 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1566750268969248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11939326544586 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1566750251929888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1575969835561984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17970450 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101892769 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354507205212160 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83349798 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.784516170324518e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1575969818522624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42560357Ïl Ì”2‹"- _wandbruntime‚9343  _runtime‚9343Óü»ò Œ"… /loss/FRt1/detection_loss‚0.5196057073265676 /loss/FBt2/detection_loss‚0.4461005766952068 % _timestamp‚1.7845161703338668e+09 1loss/SRqqt1/detection_loss‚0.7885959982670265 1loss/SRqqt2/detection_loss‚0.6522148500290911 epoch‚25 _runtime‚9343.909217412 0loss/FRt2/assignment_loss‚0.8739923504758472 2loss/SRqqt2/assignment_loss‚0.5221041098158286 /loss/FRt2/detection_loss‚1.6800032545897094 /loss/FBt1/detection_loss‚0.4054713102091374 %loss/total_loss‚8.089318391500026 trainer/global_step‚10299 _step‚848 0loss/FRt1/assignment_loss‚1.6145570184637688 2loss/SRqqt1/assignment_loss‚0.5640986483364572 1loss/FBt1/assignment_loss‚0.02043651771014418 3loss/FBt2/assignment_loss‚0.0021380495812418792Ñð*Ð"Š 0loss/FRt2/assignment_loss‚0.8739923504758472 2loss/SRqqt2/assignment_loss‚0.5221041098158286 /loss/FRt2/detection_loss‚1.6800032545897094 /loss/FBt1/detection_loss‚0.4054713102091374 %loss/total_loss‚8.089318391500026 trainer/global_step‚10299 _step‚848 0loss/FRt1/assignment_loss‚1.6145570184637688 2loss/SRqqt1/assignment_loss‚0.5640986483364572 1loss/FBt1/assignment_loss‚0.02043651771014418 3loss/FBt2/assignment_loss‚0.0021380495812418792 /loss/FRt1/detection_loss‚0.5196057073265676 /loss/FBt2/detection_loss‚0.4461005766952068 % _timestamp‚1.7845161703338668e+09 1loss/SRqqt1/detection_loss‚0.7885959982670265 1loss/SRqqt2/detection_loss‚0.6522148500290911 epoch‚25 _runtime‚9343.909217412ÐYØ$2wŽ"‚(:ì ËœöÒÚúÖË& disk.nvme0n1.out‚54297.38818359375 proc.memory.availableMB‚493) disk./.usagePercent‚75.97386804781642# disk.dm-0.out‚71116.15185546875! disk.nvme0c0n1.in‚ 162.0859375 proc.cpu.threads‚17 network.sent‚ 1.6391513e+07 network.recv‚ 2.468411e+06" disk.nvme1c1n1.in‚ 217.30859375 disk.dm-0.in‚ 379.39453125 disk.nvme0n1.in‚ 162.0859375" memory_percent‚96.990966796875 proc.memory.rssMB‚ 3023.6875* proc.memory.percent‚18.455123901367188% disk./.usageGB‚1166.3888092041016( disk.nvme1c1n1.out‚30272.77685546875& disk.nvme1n1.out‚30272.77685546875 disk.md0.out‚ 520.97265625 disk.rbd0.in‚0 cpu‚25.693667514720342 disk.nvme1n1.in‚ 217.30859375 disk.md0.in‚0 disk.dm-1.in‚0# disk.dm-1.out‚12373.23583984375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚54297.38818359375û»þž"‚(:“ ËœöÒªÁ×Ë gpu.0.gpu‚93.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚281.109' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.70299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö:ÝÐy"‚(:î ÚœöÒéêÌË network.sent‚ 1.6442112e+07 disk.rbd0.out‚0' disk.nvme0c0n1.out‚54307.6142578125 disk.nvme0n1.in‚ 162.0859375# disk.dm-1.out‚12373.31787109375 proc.cpu.threads‚17 network.recv‚ 2.47247e+06( disk./.usagePercent‚75.9739033311261 disk.nvme1n1.in‚ 217.30859375% disk.nvme1n1.out‚30274.7216796875 disk.md0.out‚ 521.30078125 disk.dm-1.in‚0 proc.memory.rssMB‚ 3023.6875! disk.nvme0c0n1.in‚ 162.0859375% disk.nvme0n1.out‚54307.6142578125" disk.nvme1c1n1.in‚ 217.30859375' disk.nvme1c1n1.out‚30274.7216796875# disk.dm-0.out‚71127.52294921875* proc.memory.percent‚18.455123901367188 cpu‚25.6943713432142% disk./.usageGB‚1166.3893508911133 disk.rbd0.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 379.39453125$ memory_percent‚96.98958396911621' proc.memory.availableMB‚ 493.2265625¢õãìž‘"‚(:“ ÚœöÒ—Ò–Ì gpu.0.gpu‚98.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚260.116' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.70533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êbSDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1595246758785024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1585968073045006 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1585967715444372 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83731653 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚9385.692025274 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1583235286167040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24701424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102357608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚344108510472704 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83731653 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1573974371675376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18055413 trainer/global_step‚10349 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1595246758785024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42756837 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12079392972288 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1583235286167040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚356187781660672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1583235269127680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42757132 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12011831081472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186099406 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845162121135924e+09 _step‚850 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1595247117248512 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11993360808356 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12079289800192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1583235269127680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102367751 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24701716 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272ÚmJ¥™"…P QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102357608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚344108510472704 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83731653 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1573974371675376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18055413 trainer/global_step‚10349 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1595246758785024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42756837 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12079392972288 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1583235286167040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚356187781660672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1583235269127680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42757132 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12011831081472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186099406 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845162121135924e+09 _step‚850 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1595247117248512 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11993360808356 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12079289800192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1583235269127680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102367751 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24701716 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83731655 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚344108491860480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102367751 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12011831081472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356187903444992 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1573974354636016 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186089261 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83731655 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12011489657344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12011489657344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1595247117248512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11993701369630 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1595246758785024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1585968073045006 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatÁ‘™f sMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1585967715444372 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83731653 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚9385.692025274 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1583235286167040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24701424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452ÒSŸuC2š"- _wandbruntime‚9386  _runtime‚9386“v'›"‰N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186116869 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1583383560669184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24703938 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚344141672130560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚851 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚10349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186106710 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83739446 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12012941187584 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1595396143386112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186116869 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102367264 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12012599756288 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24703653 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42760800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11994809757994 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102377421 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18057150 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1574121785303488 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚344141690742784 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1574121802342848 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1595396501856768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 % _timestamp‚1.7845162127234523e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1595396501856768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1586116254500096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚ 9386.29917912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356222204029440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚356222073863680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1583383543629824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18057147 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1586116612100842 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1583383560669184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83739448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12080401733120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11994469196608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344ƒ´˜'œ"ŽN REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83739448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12080401733120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11994469196608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186116869 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1583383560669184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24703938 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚344141672130560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚851 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚10349 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186106710 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83739446 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12012941187584 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1595396143386112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186116869 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102367264 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12012599756288 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24703653 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42760800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11994809757994 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102377421 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18057150 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1574121785303488 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚344141690742784 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1574121802342848 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1595396501856768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 % _timestamp‚1.7845162127234523e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1595396501856768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1586116254500096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚ 9386.29917912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356222204029440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚356222073863680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1583383543629824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18057147 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1586116612100842 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1583383560669184Ó¨^Y2"- _wandbruntime‚9386  _runtime‚9386Xr1Œ ž"„ 2loss/SRqqt1/assignment_loss‚0.5757092351756824 3loss/FBt2/assignment_loss‚0.0025807771722075046 %loss/total_loss‚8.471516255875859 trainer/global_step‚10349 1loss/SRqqt2/detection_loss‚0.7056828918154936 0loss/FBt1/detection_loss‚0.46001323564754176 epoch‚25 1loss/SRqqt2/assignment_loss‚0.572280019298651 /loss/FBt2/detection_loss‚0.6221178895370527 _runtime‚9386.305003101 /loss/FRt2/assignment_loss‚0.929621861160146 2loss/FBt1/assignment_loss‚0.009195940376425512 /loss/FRt1/detection_loss‚0.4941347246409772 /loss/FRt2/detection_loss‚1.8389113138031452 1loss/SRqqt1/detection_loss‚0.6398030086345197 % _timestamp‚1.7845162127267268e+09 _step‚852 /loss/FRt1/assignment_loss‚1.621465358614015Ij·Ÿ"‰ 1loss/SRqqt2/detection_loss‚0.7056828918154936 0loss/FBt1/detection_loss‚0.46001323564754176 epoch‚25 1loss/SRqqt2/assignment_loss‚0.572280019298651 /loss/FBt2/detection_loss‚0.6221178895370527 _runtime‚9386.305003101 /loss/FRt2/assignment_loss‚0.929621861160146 2loss/FBt1/assignment_loss‚0.009195940376425512 /loss/FRt1/detection_loss‚0.4941347246409772 /loss/FRt2/detection_loss‚1.8389113138031452 1loss/SRqqt1/detection_loss‚0.6398030086345197 % _timestamp‚1.7845162127267268e+09 _step‚852 /loss/FRt1/assignment_loss‚1.621465358614015 2loss/SRqqt1/assignment_loss‚0.5757092351756824 3loss/FBt2/assignment_loss‚0.0025807771722075046 %loss/total_loss‚8.471516255875859 trainer/global_step‚10349Ô„­uóy "‚(:î øœöÒÒ•°Ë( disk./.usagePercent‚76.0917562942244# disk./.usageGB‚1168.19868850708 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1n1.out‚30404.8017578125" disk.dm-0.out‚73015.5185546875' proc.memory.availableMB‚ 493.1953125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚56071.41845703125& disk.nvme0n1.out‚56071.41845703125 disk.md0.in‚0$ memory_percent‚96.98977470397949 cpu‚25.695666326052187 disk.nvme0n1.in‚ 165.3046875" disk.nvme1c1n1.in‚ 217.30859375' disk.nvme1c1n1.out‚30404.8017578125 network.sent‚ 1.6451052e+07 disk.dm-1.in‚0# disk.dm-1.out‚12374.07373046875! disk.nvme0c0n1.in‚ 165.3046875 disk.md0.out‚ 523.8046875 network.recv‚ 2.47989e+06 disk.nvme1n1.in‚ 217.30859375 disk.dm-0.in‚ 382.61328125 proc.memory.rssMB‚ 3023.6875* proc.memory.percent‚18.455123901367188_Ò¡"‚(:’ øœöÒæ’´Ì gpu.0.gpu‚95.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚123.77' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.25666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jèÛôv¢"‚(:ë ‡öÒ«”ÈË% disk./.usageGB‚1168.1982040405273 disk.dm-0.in‚383.875# disk.dm-1.out‚12374.29052734375 disk.rbd0.out‚0# memory_percent‚96.9897985458374( proc.memory.availableMB‚ 493.19140625 proc.memory.rssMB‚ 3023.6875 network.recv‚ 2.484528e+06) disk./.usagePercent‚76.09172473802491 disk.nvme1n1.in‚ 217.30859375# disk.nvme1n1.out‚30405.71484375 disk.md0.out‚ 524.39453125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 166.56640625 cpu‚25.69628248140504 network.sent‚ 1.6513489e+07& disk.nvme0n1.out‚56073.88818359375" disk.nvme1c1n1.in‚ 217.30859375 disk.md0.in‚0( disk.nvme0c0n1.out‚56073.88818359375* proc.memory.percent‚18.455123901367188 proc.cpu.threads‚17 disk.nvme0n1.in‚ 166.56640625% disk.nvme1c1n1.out‚30405.71484375" disk.dm-0.out‚73017.4326171875 disk.dm-1.in‚0õøî–ž£"‚(:“ ‡öÒš©¥Ì gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚268.432' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.47733333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uG«ôž¤"‚(:“ –öÒ‚­²Ë gpu.0.gpu‚57.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚285.423' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.14099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡7v¥"‚(:ë –öÒúéËË network.recv‚ 2.487699e+06 disk.nvme0n1.in‚ 166.56640625( disk.nvme1c1n1.out‚30409.44287109375 disk.md0.out‚ 526.51171875 disk.dm-0.out‚ 73031.5546875' proc.memory.availableMB‚ 493.1953125 proc.memory.rssMB‚ 3023.6875" disk.nvme1c1n1.in‚ 217.30859375 network.sent‚ 1.6519169e+07 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 166.56640625& disk.nvme0n1.out‚56088.69677734375 cpu‚25.69642069900938 disk.rbd0.in‚0( disk.nvme0c0n1.out‚56088.69677734375 disk.nvme1n1.in‚ 217.30859375& disk.nvme1n1.out‚30409.44287109375 disk.dm-0.in‚383.875$ memory_percent‚96.98977470397949* proc.memory.percent‚18.455123901367188) disk./.usagePercent‚76.09168150354687% disk./.usageGB‚1168.1975402832031 disk.dm-1.in‚0 disk.dm-1.out‚12374.41015625 disk.md0.in‚0 proc.cpu.threads‚17 ˆße2¦"- _wandbruntime‚9428  _runtime‚9428êÆHʧ"Ä ' lr-AdamW/pg1‚0.00029744625489917966 ' lr-AdamW/pg2‚0.00029744625489917966 trainer/global_step‚10399 % _timestamp‚1.7845162549874845e+09 _runtime‚9428.564019042 _step‚853=‚^‘Ϩ"É _step‚853 ' lr-AdamW/pg1‚0.00029744625489917966 ' lr-AdamW/pg2‚0.00029744625489917966 trainer/global_step‚10399 % _timestamp‚1.7845162549874845e+09 _runtime‚9428.564019042ÕÅ$RÞ2©"- _wandbruntime‚9428  _runtime‚9428pi#¹Aª"€P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1602716052994048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102852403 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1593395089632014 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42957627 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚10399 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12135126748672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12067400183296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18142116 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18142113 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12049184583038 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102852403 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84121303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1590649011274240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1602716052994048 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1602716411457536 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1590649011274240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24815799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84121303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84121305 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚357902632113152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚357902764383232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345767505364480 GADeviceStatsMonitor.on_train_batc¥QMÅh_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1593394732031380 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1581345888009616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12067058759168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12067058759168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1590648994234880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12067400183296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚9428.564103132 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102842260 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42957915 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102842260 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12135240406528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186963563 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1602716411457536 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345767523976704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186973708 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1581345905048976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1593395089632014 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42957627 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚10399 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12135126748672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12067400183296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18142116 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18142113 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12049184583038 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102852403 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84121303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1590649011274240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1602716052994048 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1602716411457536 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1590649011274240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24815799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84121303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84121305 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚357902632113152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚357902764383232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345767505364480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1593394732031380 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1581345888009616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12067058759168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12067058759168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1590648994234880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12067400183296 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚9428.564103132 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102842260 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42957915Ö‰¶™2¬"- _wandbruntime‚9429  _runtime‚9429¾¯]['­"ˆN BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345800704246784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1602865796065792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186991171 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345800685634560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186991171 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18143850 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚357937056716288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186981012 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42961915 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84129098 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12068510289408 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12068168858112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102851916 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24817777 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1590797268737024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1593543628687850 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1581493335716448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42961624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1590797285776384 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12136352469504 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1581493318677088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1593543271087104 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12068168858112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12050292971402 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84129098 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12068510289408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24818065 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102862073 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84129096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12136245110272 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12049952410016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚10399 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345800704246784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1602865796065792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186991171 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345800685634560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186991171 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18143850 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚357937056716288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186981012 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42961915 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84129098 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12068510289408 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12068168858112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102851916 QCDeviceStatsMonitor.on_train_batch_end/inazÕ¢Fctive_split_bytes.all.peak‚ 717119488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24817777 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1590797268737024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1593543628687850 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1581493335716448 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42961624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1590797285776384 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12136352469504 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1581493318677088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1593543271087104 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12068168858112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12050292971402 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84129098 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12068510289408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24818065 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102862073 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288×ýÁY2¯"- _wandbruntime‚9429  _runtime‚9429ÔçÖ °"‡ 1loss/SRqqt2/detection_loss‚0.6596658390437163 %loss/total_loss‚8.056343557242336 epoch‚25 0loss/FRt1/assignment_loss‚1.6187235729530254 0loss/FRt2/assignment_loss‚0.7839367915510507 2loss/SRqqt2/assignment_loss‚0.5605148750982498 0loss/FRt1/detection_loss‚0.43598612790885655 /loss/FRt2/detection_loss‚1.6723082990962184 trainer/global_step‚10399 /loss/FBt1/detection_loss‚0.5344173588156182 /loss/FBt2/detection_loss‚0.5506878046869812 3loss/SRqqt1/assignment_loss‚0.46317963185876526 2loss/FBt2/assignment_loss‚0.005778110834953154 1loss/SRqqt1/detection_loss‚0.7693182818223235 $ _timestamp‚1.784516255600742e+09 _step‚856 3loss/FBt1/assignment_loss‚0.0018268635725771662 _runtime‚9429.176053971ÍAr‹±"Œ 2loss/SRqqt2/assignment_loss‚0.5605148750982498 0loss/FRt1/detection_loss‚0.43598612790885655 /loss/FRt2/detection_loss‚1.6723082990962184 trainer/global_step‚10399 /loss/FBt1/detection_loss‚0.5344173588156182 /loss/FBt2/detection_loss‚0.5506878046869812 3loss/SRqqt1/assignment_loss‚0.46317963185876526 2loss/FBt2/assignment_loss‚0.005778110834953154 1loss/SRqqt1/detection_loss‚0.7693182818223235 $ _timestamp‚1.784516255600742e+09 _step‚856 3loss/FBt1/assignment_loss‚0.0018268635725771662 _runtime‚9429.176053971 1loss/SRqqt2/detection_loss‚0.6596658390437163 %loss/total_loss‚8.056343557242336 epoch‚25 0loss/FRt1/assignment_loss‚1.6187235729530254 0loss/FRt2/assignment_loss‚0.7839367915510507Ø d®ø{²"‚(:ð ¥öÒŠ¦Ë# proc.memory.rssMB‚ 3024.40234375 network.sent‚ 1.6522293e+07 disk.nvme0n1.in‚ 166.56640625 disk.nvme1n1.in‚ 217.30859375 disk.md0.in‚0" disk.dm-0.out‚73032.4443359375" disk.dm-1.out‚12375.5517578125# proc.memory.availableMB‚492.125 cpu‚25.69775063576121% disk./.usageGB‚1167.9601402282715" disk.nvme1c1n1.in‚ 217.30859375 disk.dm-0.in‚383.875 proc.cpu.threads‚17 network.recv‚ 2.491444e+06) disk./.usagePercent‚76.07621822037267 disk.md0.out‚ 527.87890625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚56090.86572265625$ memory_percent‚96.99630737304688* proc.memory.percent‚18.459486961364746& disk.nvme0n1.out‚56090.86572265625( disk.nvme1c1n1.out‚30412.11376953125& disk.nvme1n1.out‚30412.11376953125 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 166.566406259²é,ž³"‚(:“ ¥öÒ³Â¯Ë gpu.0.gpu‚83.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚285.418' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.13933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºßFB2´"- _wandbruntime‚9444  _runtime‚9444‘ý¥=µ"‡{ 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9994120597839355 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946585297584534 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949055910110474 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_1_of_3‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6614583730697632 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5872740745544434 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.300598680973053 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28054800629615784 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9799898266792297 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7516446709632874 +particle/sensitivity‚0.6415489912033081 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6236178874969482 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6568852663040161 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965057373046875 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20209871232509613 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6579967141151428 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30023306608200073 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6468060612678528 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6724718809127808 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 epoch‚25 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20420467853546143 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27946895360946655 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6736312508583069 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26708051562309265 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6723343729972839 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.726970374584198 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8619920015335083 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.551963210105896 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06773442029953003 particle/accuracy_1_of_4‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.346220999956131 5 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.99571692943573 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6801567673683167 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8903628587722778 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3056403696537018 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8363006114959717 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6224879622459412 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.059559594839811325 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48363250494003296 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987666010856628 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8760204315185547 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053608208894729614 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 trainer/global_step‚10399 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7210180163383484 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47345080971717834 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2567242681980133 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4414980411529541 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4010416865348816 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6469535231590271 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06485211104154587 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2090996950864792 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.702020525932312 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09338817745447159 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912253022193909 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.057307396084070206 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6123387217521667 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16500000655651093 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6391749382019043 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49768516421318054 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.672080397605896 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7989229559898376 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31096920371055603 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20515628159046173 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5699999928474426 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9537025690078735 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6824713945388794 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5966585874557495 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3428260087966919 'particle/f_score‚0.6555188298225403 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2568715512752533 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *loss/val_total_loss‚7.6774760425557345 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8892319798469543 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9865995049476624 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7542370557785034 particle/accuracy_4_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6069257259368896 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8764740228652954 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3675621449947357 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6679214835166931 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.042274050414562225 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28656741976737976 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.45960158109664917 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28898248076438904 particle/accuracy_1_of_1‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21873876452445984 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04840986058115959 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2408941090106964 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5945368409156799 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8552051782608032 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06536286324262619 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6116321086883545 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9899949431419373 particle/accuracy_3_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9095179438591003 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8979214429855347 particle/accuracy_2_of_3‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28898248076438904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6778033971786499 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148655891418 particle/accuracy_1_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 jet/accuracy_3_of_5‚1 _runtime‚9444.953116816 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7520104646682739 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.517755925655365 particle/accuracy_2_of_4‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9852859973907471 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26827797293663025 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7453703284263611 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6730772852897644 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8788738250732422 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6383111476898193 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3005370795726776 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27237048745155334 *jet/accuracy_2_of_4‚0.8084000945091248 +jet/accuracy_4_of_4‚0.01670061983168125 +particle/specificity‚0.9136444330215454 _step‚857 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7453703284263611 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 % _timestamp‚1.7845162713770201e+09 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7875977754592896 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28842732310295105 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930979609489441 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.722062349319458 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.09274070709943771 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995768129825592 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11086325347423553 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21585212647914886 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6190522313117981 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6463814377784729 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6908077597618103 +jet/accuracy_2_of_2‚0.15832142531871796 *jet/accuracy_2_of_3‚0.5720979571342468 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 *jet/accuracy_1_of_4‚0.9816489219665527 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3644337058067322 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.679764449596405 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7475977540016174 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19073395431041718 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29834529757499695 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6116321086883545 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06683892011642456 6validation_average_jet_accuracy‚0.3824550211429596 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2552095055580139 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3453218340873718 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 jet/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981386065483093 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8363006114959717 *jet/accuracy_1_of_3‚0.9246662855148315 +jet/accuracy_3_of_3‚0.07708815485239029 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4880976974964142 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22897693514823914 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6913232207298279 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7551754713058472 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9520063996315002 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9223443269729614 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.059559594839811325 particle/accuracy_2_of_2‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912398457527161 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_5‚1ÉÍWR(¶"Œ{ =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6736312508583069 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26708051562309265 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6723343729972839 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.726970374584198 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8619920015335083 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.551963210105896 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06773442029953003 particle/accuracy_1_of_4‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.346220999956131 5 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.99571692943573 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6801567673683167 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8903628587722778 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3056403696537018 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8363006114959717 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6224879622459412 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.059559594839811325 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48363250494003296 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987666010856628 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8760204315185547 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.053608208894729614 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 trainer/global_step‚10399 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7210180163383484 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47345080971717834 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2567242681980133 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4414980411529541 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4010416865348816 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6469535231590271 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06485211104154587 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2090996950864792 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.702020525932312 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09338817745447159 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912253022193909 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.057307396084070206 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6123387217521667 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16500000655651093 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6391749382019043 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49768516421318054 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.672080397605896 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7989229559898376 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31096920371055603 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20515628159046173 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5699999928474426 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9537025690078735 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6824713945388794 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5966585874557495 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3428260087966919 'particle/f_score‚0.6555188298225403 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2568715512752533 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *loss/val_total_loss‚7.6774760425557345 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8892319798469543 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9865995049476624 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7542370557785034 particle/accuracy_4_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6069257259368896 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8764740228652954 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3675621449947357 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6679214835166931 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.042274050414562225 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28656741976737976 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.45960158109664917 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28898248076438904 particle/accuracy_1_of_1‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21873876452445984 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04840986058115959 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2408941090106964 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5945368409156799 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8552051782608032 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06536286324262619 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6116321086883545 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9899949431419373 particle/accuracy_3_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9095179438591003 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8979214429855347 particle/accuracy_2_of_3‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28898248076438904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6778033971786499 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148655891418 particle/accuracy_1_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 jet/accuracy_3_of_5‚1 _runtime‚9444.953116816 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7520104646682739 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.517755925655365 particle/accuracy_2_of_4‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9852859973907471 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26827797293663025 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7453703284263611 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6730772852897644 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8788738250732422 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6383111476898193 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3005370795726776 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27237048745155334 *jet/accuracy_2_of_4‚0.8084000945091248 +jet/accuracy_4_of_4‚0.01670061983168125 +particle/specificity‚0.9136444330215454 _step‚857 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7453703284263611 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 % _timestamp‚1.7845162713770201e+09 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7875977754592896 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28842732310295105 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930979609489441 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.722062349319458 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.09274070709943771 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995768129825592 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11086325347423553 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21585212647914886 9"Purity/0FRt*SRqqt1FBt/SRý›¼Ç@qqt_purity‚0.6190522313117981 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6463814377784729 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6908077597618103 +jet/accuracy_2_of_2‚0.15832142531871796 *jet/accuracy_2_of_3‚0.5720979571342468 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 *jet/accuracy_1_of_4‚0.9816489219665527 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3644337058067322 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.679764449596405 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7475977540016174 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19073395431041718 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29834529757499695 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6116321086883545 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06683892011642456 6validation_average_jet_accuracy‚0.3824550211429596 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2552095055580139 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3453218340873718 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 jet/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981386065483093 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8363006114959717 *jet/accuracy_1_of_3‚0.9246662855148315 +jet/accuracy_3_of_3‚0.07708815485239029 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4880976974964142 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22897693514823914 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6913232207298279 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7551754713058472 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9520063996315002 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9223443269729614 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.059559594839811325 particle/accuracy_2_of_2‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912398457527161 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9994120597839355 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946585297584534 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9949055910110474 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_1_of_3‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6614583730697632 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5872740745544434 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.300598680973053 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28054800629615784 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9799898266792297 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7516446709632874 +particle/sensitivity‚0.6415489912033081 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6236178874969482 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6568852663040161 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965057373046875 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20209871232509613 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6579967141151428 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30023306608200073 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6468060612678528 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6724718809127808 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 epoch‚25 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20420467853546143 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.27946895360946655 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75Ù†c¦w·"‚(:ì ´öÒ‘“ Ë) proc.memory.percent‚18.45989227294922 cpu‚25.700157743336696 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 1.658471e+07 network.recv‚ 2.495591e+06 disk.md0.in‚0 disk.md0.out‚ 528.76171875 disk.nvme1n1.in‚ 217.30859375% disk.nvme1n1.out‚30433.7626953125$ memory_percent‚96.99821472167969 proc.memory.rssMB‚ 3024.46875) disk./.usagePercent‚75.98941034572634% disk./.usageGB‚1166.6274223327637 disk.dm-0.in‚383.875# disk.dm-0.out‚73039.61865234375" disk.dm-1.out‚12395.6025390625" disk.nvme0c0n1.in‚ 166.56640625' disk.nvme1c1n1.out‚30433.7626953125$ proc.memory.availableMB‚491.8125( disk.nvme0c0n1.out‚56098.24755859375 disk.nvme0n1.in‚ 166.56640625& disk.nvme0n1.out‚56098.24755859375" disk.nvme1c1n1.in‚ 217.30859375²ä6òŸ¸"‚(:” ´öÒÝ›¥Ë gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚123.023' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.007666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ô»On¹"‚(:ã ÃöÒñ¡§Ë" disk.nvme0c0n1.in‚ 166.56640625) proc.memory.percent‚18.45989227294922 cpu‚25.699840761135793" disk.nvme1c1n1.in‚ 217.30859375 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 56099.3359375$ memory_percent‚96.99511528015137 network.sent‚ 1.6643537e+07 network.recv‚ 2.500239e+06 disk.dm-0.in‚383.875 proc.cpu.threads‚17% disk./.usageGB‚1166.6272506713867 disk.nvme0n1.in‚ 166.56640625" disk.nvme0n1.out‚ 56099.3359375( disk.nvme1c1n1.out‚30949.44580078125 disk.nvme1n1.in‚ 217.30859375 disk.md0.out‚529.125' proc.memory.availableMB‚ 492.3203125 proc.memory.rssMB‚ 3024.46875( disk./.usagePercent‚75.9893991643958& disk.nvme1n1.out‚30949.44580078125# disk.dm-0.out‚73040.52490234375! disk.dm-1.out‚12910.681640625 disk.rbd0.in‚0øRÆžº"‚(:“ ÃöÒÁ¾ïË gpu.0.gpu‚82.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚280.278' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.42600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P ;Er»"‚(:ç ÒöÒ®¬£Ë proc.cpu.threads‚17 network.sent‚ 1.6648358e+07 disk.nvme1n1.in‚ 217.30859375 disk.dm-0.in‚383.875# disk.dm-0.out‚73059.23876953125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 166.56640625" disk.nvme1c1n1.in‚ 217.30859375) disk./.usagePercent‚75.81238503604423# disk.nvme1n1.out‚30950.59765625 disk.md0.out‚ 529.5078125! disk.dm-1.out‚12910.744140625 disk.rbd0.out‚0) proc.memory.percent‚18.45989227294922 cpu‚25.699525272939756% disk./.usageGB‚1163.9096412658691 disk.dm-1.in‚0( disk.nvme0c0n1.out‚56117.77978515625% disk.nvme1c1n1.out‚30950.59765625$ proc.memory.availableMB‚492.3125 network.recv‚ 2.503241e+06 disk.md0.in‚0" disk.nvme0c0n1.in‚ 166.56640625& disk.nvme0n1.out‚56117.77978515625$ memory_percent‚96.99516296386719 proc.memory.rssMB‚ 3024.46875 Eþfž¼"‚(:“ ÒöÒî‚åË gpu.0.gpu‚55.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚291.192' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.06400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖȇÓ2½"- _wandbruntime‚9486  _runtime‚9486Nì¶É¾"à $ _timestamp‚1.784516313037614e+09 _runtime‚9486.614049488 _step‚858 ' lr-AdamW/pg1‚0.00029498550726561905 ' lr-AdamW/pg2‚0.00029498550726561905 trainer/global_step‚10449ÙÎò3ο"È _runtime‚9486.614049488 _step‚858 ' lr-AdamW/pg1‚0.00029498550726561905 ' lr-AdamW/pg2‚0.00029498550726561905 trainer/global_step‚10449 $ _timestamp‚1.784516313037614e+09Ú¥%Ð2À"- _wandbruntime‚9486  _runtime‚9486%T›(Á"ÿO aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1600005080257024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84633763 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103418549 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1602763666076143 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12143287120896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103418549 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1612148008914432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188052312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84633761 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18232932 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784516313038279e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103408406 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1600005063217664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188042167 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1602763308475509 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚347449116143104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1612148008914432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359660944274432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _runtime‚9486.614114358 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1612148367377920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43198200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12142945696768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18232929 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43197919 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚359660824587264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347449097530880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12211828131328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24964990 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚859 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84633763 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103408406 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84633761 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12143287120896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1600005063217664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12142945696768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12124971645343 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10449 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1590638694430800 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272µqÄ« (Â"„P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚347449116143104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1612148008914432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359660944274432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _runtime‚9486.614114358 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1612148367377920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43198200 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12142945696768 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18232929 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43197919 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚359660824587264 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347449097530880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12211828131328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24964990 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _step‚859 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84633763 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103408406 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84633761 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12143287120896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1600005063217664 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12142945696768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12124971645343 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10449 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1590638694430800 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1600005080257024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84633763 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103418549 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1602763666076143 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12143287120896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103418549 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1612148008914432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188052312 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84633761 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18232932 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784516313038279e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103408406 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1600005063217664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188042167 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1602763308475509ÛYïˆ2Ã"- _wandbruntime‚9487  _runtime‚9487¼"È5éÄ"‰N OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 trainer/global_step‚10449 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188069775 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12125739472321 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359695119723008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18234663 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REô¢1•&DeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24967321 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18234666 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1602912205131979 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1612297751986176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24967600 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103418062 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12144397227008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12144055795712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1600153354759168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84641554 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43201984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚347482296413184 _runtime‚9487.226256821 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1600153354759168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12212940892672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188059616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 % _timestamp‚1.7845163136504624e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188059616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1600153337719808 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84641556 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1590786125098272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188069775 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1612297393515520 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12144055795712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84641554 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1600153337719808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84641556 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359695237305856 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12212841922048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103428219 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12144397227008 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103428219 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43202266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1590786108058912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚860 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103418062 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0‘.S'Å"ŽN OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43202266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1590786108058912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚860 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103418062 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 trainer/global_step‚10449 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188069775 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12125739472321 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359695119723008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18234663 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24967321 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18234666 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1602912205131979 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1612297751986176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24967600 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103418062 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12144397227008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12144055795712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1600153354759168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84641554 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43201984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚347482296413184 _runtime‚9487.226256821 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1600153354759168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12212940892672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188059616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 % _timestamp‚1.7845163136504624e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188059616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1600153337719808 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84641556 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1590786125098272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188069775 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1612297393515520 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12144055795712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84641554 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1600153337719808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84641556 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359695237305856 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12212841922048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103428219 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12144397227008 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103428219 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452ÜÀDW2Æ"- _wandbruntime‚9487  _runtime‚9487ksF¦ Ç"‡ /loss/FRt1/assignment_loss‚1.622070918461228 /loss/FRt1/detection_loss‚0.4883675182842968 1loss/SRqqt1/detection_loss‚0.6669603711036121 trainer/global_step‚10449 0loss/SRqqt2/detection_loss‚0.621771056928149 0loss/FBt1/detection_loss‚0.40268459724802197 epoch‚26 0loss/FRt2/assignment_loss‚0.7010461288563749 3loss/SRqqt2/assignment_loss‚0.48747980237183225 3loss/FBt2/assignment_loss‚0.0008058030807802652 /loss/FRt2/detection_loss‚1.4423731303971974 /loss/FBt2/detection_loss‚0.5098072750305829 % _timestamp‚1.7845163136514392e+09 _step‚861 2loss/SRqqt1/assignment_loss‚0.5733555564430228 3loss/FBt1/assignment_loss‚7.504526464134692e-05 %loss/total_loss‚7.516797203469739 _runtime‚9487.226762762DPMÈ"Œ epoch‚26 0loss/FRt2/assignment_loss‚0.7010461288563749 3loss/SRqqt2/assignment_loss‚0.48747980237183225 3loss/FBt2/assignment_loss‚0.0008058030807802652 /loss/FRt2/detection_loss‚1.4423731303971974 /loss/FBt2/detection_loss‚0.5098072750305829 % _timestamp‚1.7845163136514392e+09 _step‚861 2loss/SRqqt1/assignment_loss‚0.5733555564430228 3loss/FBt1/assignment_loss‚7.504526464134692e-05 %loss/total_loss‚7.516797203469739 _runtime‚9487.226762762 /loss/FRt1/assignment_loss‚1.622070918461228 /loss/FRt1/detection_loss‚0.4883675182842968 1loss/SRqqt1/detection_loss‚0.6669603711036121 trainer/global_step‚10449 0loss/SRqqt2/detection_loss‚0.621771056928149 0loss/FBt1/detection_loss‚0.40268459724802197ÝŠDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12198514798592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349108111034880 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1598010227804400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103903201 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85023411 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12267600112128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1607418788324864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43399408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12198856222720 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1619617303123456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12198856222720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361375835028480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103903201 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12267705381376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1610190325062517 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1619617303123456 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12198856222720 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361375835028480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103903201 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12267705381376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1610190325062517 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12198514798592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349108111034880 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1598010227804400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103903201 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85023411 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12267600112128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ß‚ãV2Õ"- _wandbruntime‚9529  _runtime‚9529LûïŽ'Ö"‰N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18321363 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1607567079866368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25081723 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚9529.188447602 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188944077 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1598157658471872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12199966328832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1619767046195200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1619767046195200 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188944077 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12181563247115 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1607567062827008 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349141291304960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85031206 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1619766687724544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18321366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25081994 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361410127035904 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1598157641432512 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚10499 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103902714 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12199624897536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85031204 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12268817118720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12181222685729 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1607567079866368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85031206 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103902714 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103912871 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43403360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188933918 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103912871 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1610338864118241 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43403086 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188933918 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188933918 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188944077 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12181563247115 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1607567062827008 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349141291304960 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85031206 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1619766687724544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18321366 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25081994 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361410127035904 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1598157641432512 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚10499 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103902714 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12199624897536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85031204 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12268817118720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12181222685729 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1607567079866368 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85031206 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103902714 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103912871 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43403360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188933918 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103912871 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1610338864118241 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43403086 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0à](2Ø"- _wandbruntime‚9529  _runtime‚9529x@ºýÙ"ˆ 3loss/SRqqt1/assignment_loss‚0.42808025733840904 3loss/SRqqt2/assignment_loss‚0.48061543303565907 1loss/SRqqt2/detection_loss‚0.6800506030069664 /loss/FBt2/detection_loss‚0.7317232528834471 3loss/FBt2/assignment_loss‚0.0023719427767511818 0loss/FRt1/detection_loss‚0.48857152934875697 .loss/FRt2/detection_loss‚1.629404848074417 _step‚865 epoch‚26 0loss/FRt1/assignment_loss‚1.5491473046632913 0loss/FRt2/assignment_loss‚0.8673284262114641 2loss/FBt1/assignment_loss‚0.004787850743872416 /loss/FBt1/detection_loss‚0.3220705231582677 % _timestamp‚1.7845163556143875e+09 1loss/SRqqt1/detection_loss‚0.6073049650787232 trainer/global_step‚10499 _runtime‚9529.189638954 %loss/total_loss‚7.791456936320025qYOüóÚ" 2loss/FBt1/assignment_loss‚0.004787850743872416 /loss/FBt1/detection_loss‚0.3220705231582677 % _timestamp‚1.7845163556143875e+09 1loss/SRqqt1/detection_loss‚0.6073049650787232 trainer/global_step‚10499 _runtime‚°¢¨& 9529.189638954 %loss/total_loss‚7.791456936320025 3loss/SRqqt1/assignment_loss‚0.42808025733840904 3loss/SRqqt2/assignment_loss‚0.48061543303565907 1loss/SRqqt2/detection_loss‚0.6800506030069664 /loss/FBt2/detection_loss‚0.7317232528834471 3loss/FBt2/assignment_loss‚0.0023719427767511818 0loss/FRt1/detection_loss‚0.48857152934875697 .loss/FRt2/detection_loss‚1.629404848074417 _step‚865 epoch‚26 0loss/FRt1/assignment_loss‚1.5491473046632913 0loss/FRt2/assignment_loss‚0.8673284262114641á}²»úqÛ"‚(:æ ŽžöÒ°ÄË" disk.nvme0c0n1.in‚ 166.56640625 disk.nvme0n1.in‚ 166.56640625 disk.dm-0.in‚383.875# disk.dm-0.out‚73319.01708984375" disk.dm-1.out‚12911.3720703125 proc.memory.rssMB‚ 3024.46875( disk.nvme1c1n1.out‚31035.03466796875 disk.md0.out‚ 531.015625) proc.memory.percent‚18.45989227294922 proc.cpu.threads‚17 disk.nvme1n1.in‚ 217.30859375 network.sent‚ 1.672259e+07% disk./.usageGB‚1161.4301643371582$ disk.nvme0c0n1.out‚ 56297.0234375" disk.nvme0n1.out‚ 56297.0234375" disk.nvme1c1n1.in‚ 217.30859375 disk.dm-1.in‚0 disk.rbd0.in‚0( disk./.usagePercent‚75.6508818978771 disk.md0.in‚0 disk.rbd0.out‚0$ memory_percent‚96.99814319610596( proc.memory.availableMB‚ 491.82421875 cpu‚25.69821875453399& disk.nvme1n1.out‚31035.03466796875 network.recv‚ 2.516974e+06”u4óžÜ"‚(:“ ŽžöÒ’À³Ì gpu.0.gpu‚58.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚285.232' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.07733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cVösÝ"‚(:è žöÒÔ¯ÅË disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 166.56640625 disk.nvme0n1.in‚ 166.56640625$ memory_percent‚96.99811935424805 network.recv‚ 2.521428e+06 disk.dm-0.in‚383.875" disk.dm-0.out‚73352.2333984375 disk.dm-1.in‚0% disk.nvme0n1.out‚56330.7041015625& proc.memory.availableMB‚ 491.828125 proc.memory.rssMB‚ 3024.46875 proc.cpu.threads‚17( disk./.usagePercent‚75.7725929169561" disk.nvme1c1n1.in‚ 217.30859375% disk.nvme1n1.out‚31035.8115234375 disk.md0.in‚0 disk.md0.out‚ 531.515625 network.sent‚ 1.6786946e+07' disk.nvme1c1n1.out‚31035.8115234375' disk.nvme0c0n1.out‚56330.7041015625) proc.memory.percent‚18.45989227294922 cpu‚25.69790806247391% disk./.usageGB‚1163.2987327575684 disk.nvme1n1.in‚ 217.30859375" disk.dm-1.out‚12911.5517578125rÉ,ÔžÞ"‚(:“ žöÒÚ¿ˆÌ gpu.0.gpu‚54.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚208.949' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚69.64966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š~§<Ÿß"‚(:” ¬žöÒØó³Ë gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚125.148' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.715999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú~¬wà"‚(:ì ¬žöÒÿÏ‹Ì disk.rbd0.in‚0 proc.memory.rssMB‚ 3024.46875 cpu‚25.697595295847414 proc.cpu.threads‚17# disk.dm-0.out‚75256.83544921875 disk.nvme0n1.in‚ 166.56640625" disk.nvme1c1n1.in‚ 217.30859375( disk.nvme1c1n1.out‚31038.57275390625 disk.nvme1n1.in‚ 217.30859375 disk.md0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚58233.5869140625 disk.md0.out‚ 531.96875 disk.dm-0.in‚383.875 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 166.56640625) proc.memory.percent‚18.45989227294922) disk./.usagePercent‚75.83076540411653% disk.nvme0n1.out‚58233.5869140625& disk.nvme1n1.out‚31038.57275390625" disk.dm-1.out‚12911.6416015625$ memory_percent‚96.99811935424805& proc.memory.availableMB‚ 491.828125 network.sent‚ 1.6791814e+07 network.recv‚ 2.524837e+06% disk./.usageGB‚1164.1918258666992ûg2á"- _wandbruntime‚9570  _runtime‚9570 °Èâ" % _timestamp‚1.7845163973485692e+09 _runtime‚9570.924160168 _step‚866 & lr-AdamW/pg1‚0.0002900665321543787 & lr-AdamW/pg2‚0.0002900665321543787 trainer/global_step‚10549[7÷éÍã"Ç & lr-AdamW/pg1‚0.0002900665321543787 & lr-AdamW/pg2‚0.0002900665321543787 trainer/global_step‚10549 % _timestamp‚1.7845163973485692e+09 _runtime‚9570.924160168 _step‚866âgˆÌÚ2ä"- _wandbruntime‚9570  _runtime‚9570¦žÛ#(å"€P WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1617617699250159 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18406329 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189790771 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12254083900416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12323462255104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1627086597332480 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1614832530471424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12323567524352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25194004 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12254425324544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1617617341649525 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12235597510885 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1614832530471424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25193706 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚363090586793984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189800916 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1627086597332480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85413063 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12254083900416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104387853 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1627086955795968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12254425324544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43600336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚363090710675456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1614832513432064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚350767143151104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18406332 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104377710I— (æ"…P LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1614832530471424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25193706 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚363090586793984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189800916 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1627086597332480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85413063 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12254083900416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104387853 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1627086955795968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12254425324544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43600336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚363090710675456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1614832513432064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚350767143151104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18406332 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104377710 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1617617699250159 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18406329 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189790771 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12254083900416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12323462255104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1627086597332480 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1614832530471424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12323567524352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25194004 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12254425324544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1617617341649525 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12235597510885ãÃô5‚2ç"- _wandbruntime‚9571  _runtime‚9571 Ü œè"‰N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104397523 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18408063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25196274 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85420856 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85420854 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚10549 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1617766238305995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚868 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189808220 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43604051 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚350800323421184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1614980804973568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85420856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25195988 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784516397959972e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12255535430656 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1627235981933568 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189808220 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85420854 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104387366 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12324579074560 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1617765880705249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12255535430656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363125007757824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚350800304808960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189818379 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1614980787934208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189818379 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18408066 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1605529174806112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚9571.535733181#V£‰'é"ŽN ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43604340 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85420854 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚10549 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1617766238305995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚868 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189808220 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43604051 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚350800323421184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1614980804973568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85420856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25195988 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784516397959972e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12255535430656 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1627235981933568 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189808220 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85420854 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104387366 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12324579074560 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1617765880705249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12255535430656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363125007757824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚350800304808960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189818379 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1614980787934208 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189818379 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18408066 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1605529174806112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚9571.535733181 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104397523 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18408063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25196274 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85420856 Q§Íó"Ç _runtime‚9614.540202009 _step‚870 & lr-AdamW/pg1‚0.0002876086206132786 & lr-AdamW/pg2‚0.0002876086206132786 trainer/global_step‚10599 % _timestamp‚1.7845164409646406e+09æʪ2ô"- _wandbruntime‚9614  _runtime‚9614#9͸(õ"þO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104862362 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1622246255578624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7845164409653845e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190675218 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1612753294551600 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12291421285567 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚871 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12379406177792 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1625044715837167 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12309653002240 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1634555891541504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25308442 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18493029 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1634556250004992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92686336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10599 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104872505 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18493032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚352426156655104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85802711 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1622246238539264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12291080724293 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104872505 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85802713 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12309994426368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43801471 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚352426138042880 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12309653002240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1612753277512240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190665073 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104862362 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111298560 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚ 9614.54086667 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1625044358236533 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1634555891541504 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12309994426368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25308718 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43801750 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85802711 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85802713 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364805451534336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634ÞLÐ"ö"ƒP GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10599 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104872505 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18493032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚352426156655104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85802711 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1622246238539264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12291080724293 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104872505 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85802713 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12309994426368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43801471 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚352426138042880 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12309653002240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1612753277512240 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190665073 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104862362 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111298560 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚ 9614.54086667 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1625044358236533 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1634555891541504 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12309994426368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25308718 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43801750 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85802711 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85802713 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364805451534336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104862362 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1622246255578624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7845164409653845e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190675218 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1612753294551600 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12291421285567 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚871 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12379406177792 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1625044715837167 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12309653002240 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1634555891541504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25308442 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18493029 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1634556250004992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92686336ç=±u÷"‚(:ê ÙžöÒè£ÍË$ disk./.usageGB‚1166.669303894043 disk.md0.in‚0 disk.dm-0.in‚383.875" disk.nvme0c0n1.in‚ 166.56640625 disk.nvme1n1.in‚ 217.30859375$ memory_percent‚97.00007438659668 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922# disk.dm-0.out‚78046.91748046875" disk.dm-1.out‚12912.0888671875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 166.56640625 network.sent‚ 1.6876224e+07) disk./.usagePercent‚75.99213834190152 disk.dm-1.in‚0& disk.nvme0c0n1.out‚60739.341796875$ disk.nvme0n1.out‚60739.341796875( disk.nvme1c1n1.out‚31326.68505859375' proc.memory.availableMB‚ 491.5078125 proc.cpu.threads‚17 network.recv‚ 2.535914e+06 disk.md0.out‚ 533.546875 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 217.30859375& disk.nvme1n1.out‚31326.68505859375 cpu‚25.69677862602239¸ó‹cžø"‚(:“ ÙžöÒëêË gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚264.363' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.12100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q5†ü2ù"- _wandbruntime‚9615  _runtime‚9615ñ Ï 'ú"‡N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12311104532480 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚352459336925184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 # _timestamp‚1.78451644159046e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104872018 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43805766 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104882175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12310763101184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190692681 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43805494 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104872018 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1612900725219072 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1622394530080768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190692681 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1625193254893003 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18494763 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85810504 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85810504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364839864046080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1634705634613248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25311000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1625192897292257 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190682522 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1634705276142592 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12292529673931#ý¨c'û"ŒN XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364839748560384 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364839864046080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1634705634613248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25311000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1625192897292257 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190682522 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1634705276142592 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12292529673931 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12311104532480 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚352459336925184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 # _timestamp‚1.78451644159046e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104872018 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43805766 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104882175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12310763101184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190692681 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43805494 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104872018 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1612900725219072 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1622394530080768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190692681 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1625193254893003 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18494763 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85810504 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85810504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86192363 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚310 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86192361 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1642025544214016 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1629659980685824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1629659963646464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191549520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44003424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1642025185750528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12435106420736 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1629659980685824 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1632471732424175 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12365563528192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366520390237696 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1620124810885840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44003111 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191539375 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354085151546880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105357157 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18579729 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12346904498975 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366520257967616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12365222104064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105357157 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105347014 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1632471374823541 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _step‚875 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1629659963646464 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18579732 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12435220078592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1642025185750528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12365563528192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 $ _timestamp‚1.784516485260636e+09 _runtime‚9658.836114077¬"? (ˆ#„P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12365563528192 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366520390237696 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1620124810885840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44003111 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191539375 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354085151546880 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105357157 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18579729 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12346904498975 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366520257967616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12365222104064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105357157 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105347014 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1632471374823541 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _step‚875 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1629659963646464 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18579732 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12435220078592 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1642025185750528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12365563528192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 $ _timestamp‚1.784516485260636e+09 _runtime‚9658.836114077 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1620124827925200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚313 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1642025544214016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25423382 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12365222104064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚354085170159104 trainer/global_step‚10649 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86192363 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚310 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86192361 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1642025544214016 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1629659980685824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1629659963646464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191549520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44003424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1642025185750528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12435106420736 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1629659980685824 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1632471732424175 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344ëpÿ2‰#- _wandbruntime‚9659  _runtime‚96594…ë'Š#ŠN NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86200156 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44007428 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚9659.464139514 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191566983 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191556824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105356670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25425962 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12436333478912 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1632619913879265 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1629808255187968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18581466 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191556824 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1642174570351616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44007127 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105356670 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12366332203008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191566983 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1632620271480011 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354118331816960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1629808255187968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12366673634304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1620272258592672 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366554547450880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1642174928822272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ö’H¬\ ‹#N \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1629808255187968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18581466 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191556824 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_ÿƒµ}¹bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1642174570351616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44007127 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105356670 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12366332203008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191566983 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1632620271480011 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354118331816960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1629808255187968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12366673634304 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1620272258592672 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366554547450880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1642174928822272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86200156 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44007428 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚9659.464139514 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191566983 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191556824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105356670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25425962 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12436333478912 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1632619913879265 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ìŸéTÏ2Œ#- _wandbruntime‚9659  _runtime‚96595¾^©#€ 1loss/SRqqt2/detection_loss‚0.6625318495019841 /loss/FBt2/detection_loss‚0.5248052128020446 _step‚877 /loss/FRt1/detection_loss‚0.4842313263953504 /loss/FRt2/detection_loss‚1.6551642818973125 .loss/FBt1/detection_loss‚0.522547193571971 0loss/FRt1/assignment_loss‚1.6151813941709163 1loss/FBt2/assignment_loss‚0.02375228884533248 $loss/total_loss‚8.01553196932877 epoch‚26 trainer/global_step‚10649 $ _timestamp‚1.784516485889477e+09 0loss/FRt2/assignment_loss‚0.7184337205323742 2loss/SRqqt1/assignment_loss‚0.5268225180798685 2loss/SRqqt2/assignment_loss‚0.5593754795283259 2loss/FBt1/assignment_loss‚0.006768593081061659 0loss/SRqqt1/detection_loss‚0.715918110922228 _runtime‚9659.4647635969 » Ž#… 0loss/FRt2/assignment_loss‚0.7184337205323742 2loss/SRqqt1/assignment_loss‚0.5268225180798685 2loss/SRqqt2/assignment_loss‚0.5593754795283259 2loss/FBt1/assignment_loss‚0.006768593081061659 0loss/SRqqt1/detection_loss‚0.715918110922228 _runtime‚9659.464763596 1loss/SRqqt2/detection_loss‚0.6625318495019841 /loss/FBt2/detection_loss‚0.5248052128020446 _step‚877 /loss/FRt1/detection_loss‚0.4842313263953504 /loss/FRt2/detection_loss‚1.6551642818973125 .loss/FBt1/detection_loss‚0.522547193571971 0loss/FRt1/assignment_loss‚1.6151813941709163 1loss/FBt2/assignment_loss‚0.02375228884533248 $loss/total_loss‚8.01553196932877 epoch‚26 trainer/global_step‚10649 $ _timestamp‚1.784516485889477e+09í)‘{#‚(:𠆟öÒÈó±Ë disk.rbd0.in‚0' proc.memory.availableMB‚ 491.6953125& disk.nvme0n1.out‚61705.86669921875( disk.nvme1c1n1.out‚31496.42041015625 disk.dm-0.in‚383.875! disk.dm-0.out‚79181.013671875 disk.dm-1.in‚0( disk.nvme0c0n1.out‚61705.86669921875 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922 network.sent‚ 1.6972095e+07 network.recv‚ 2.547293e+06) disk./.usagePercent‚75.98914348463771$ memory_percent‚96.99892997741699 cpu‚25.695554642691594 proc.cpu.threads‚17% disk./.usageGB‚1166.6233253479004 disk.nvme0n1.in‚ 166.56640625" disk.nvme1c1n1.in‚ 217.30859375 disk.nvme1n1.in‚ 217.30859375" disk.dm-1.out‚12912.3642578125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 166.56640625& disk.nvme1n1.out‚31496.42041015625 disk.md0.in‚0 disk.md0.out‚ 534.42578125C¡ÝŽž#‚(:“ †ŸöÒàøŸÌ gpu.0.gpu‚41.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚278.986' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.99533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ý/ìu‘#‚(:ê •ŸöÒ‡—´Ë& disk.nvme0n1.out‚61707.00341796875" disk.nvme1c1n1.in‚ 217.30859375 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 166.56640625# disk.dm-1.out‚12912.56201171875$ memory_percent‚96.99892997741699' proc.memory.availableMB‚ 491.6953125 proc.memory.rssMB‚ 3024.46875 proc.cpu.threads‚17' disk.nvme1c1n1.out‚31497.3056640625% disk.nvme1n1.out‚31497.3056640625% disk./.usageGB‚1166.6227760314941( disk.nvme0c0n1.out‚61707.00341796875 network.sent‚ 1.703465e+07 disk.md0.out‚ 534.98828125 disk.dm-0.in‚383.875! disk.dm-0.out‚79181.634765625 disk.dm-1.in‚0) proc.memory.percent‚18.45989227294922 cpu‚25.69517448044175 disk.nvme1n1.in‚ 217.30859375 network.recv‚ 2.551846e+06& disk./.usagePercent‚75.98910770438 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 166.56640625æ‚ù#œ’#‚(:‘ •ŸöÒÕ¤ƒÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚279.082' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.02733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`*[v“#‚(:ë ¤ŸöÒºÙ±Ë network.recv‚ 2.554871e+06 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 166.56640625( disk.nvme0c0n1.out‚61779.09521484375" disk.nvme1c1n1.in‚ 217.30859375' proc.memory.availableMB‚ 491.4453125) proc.memory.percent‚18.45989227294922 proc.cpu.threads‚17# disk.dm-1.out‚12912.66357421875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 166.56640625 disk.nvme1n1.in‚ 217.30859375& disk.nvme1n1.out‚31498.75341796875 cpu‚25.69482144836522 network.sent‚ 1.7040283e+07) disk./.usagePercent‚76.00877690715134$ disk./.usageGB‚1166.924747467041( disk.nvme1c1n1.out‚31498.75341796875# disk.dm-0.out‚79254.48974609375 disk.rbd0.in‚0& disk.nvme0n1.out‚61779.09521484375 disk.md0.in‚0 disk.md0.out‚535.25 disk.dm-0.in‚383.875$ memory_percent‚97.00045585632324 proc.memory.rssMB‚ 3024.46875Ëòˆ”#‚(:’ ¤ŸöÒ¶íË gpu.0.gpu‚44.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚292.06' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.35333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XC*(2•#- _wandbruntime‚9703  _runtime‚9703m‹UöÉ–#à ' lr-AdamW/pg1‚0.00028269673888530546 ' lr-AdamW/pg2‚0.00028269673888530546 trainer/global_step‚10699 $ _timestamp‚1.784516529725754e+09 _runtime‚9703.301343603 _step‚878ÓŠqΗ#È ' lr-AdamW/pg2‚0.00028269673888530546 trainer/global_step‚10699 $ _timestamp‚1.784516529725754e+09 _runtime‚9703.301343603 _step‚878 ' lr-AdamW/pg1‚0.00028269673888530546îÚ·fV2˜#- _wandbruntime‚9703  _runtime‚9703ºó (™#€P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1637073705793024 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25538725 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12402387712383 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86582013 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1649494838423040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 192413677 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1637073688753664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86582013 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚355744183663104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105831666 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105841809 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1649494479959552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44205154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86582011 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12421132630016 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12421132630016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12490917623296 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105841809 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚368235214944256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1627496361298800 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105831666 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7845165297268083e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12420791205888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192423822 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12491031281152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1639898391410549 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18666432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1637073688753664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚879 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1649494838423040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1639898749011183 _runtime‚9703.302451009 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1637073705793024 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12420791205888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86582011 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0rõ¬V&š#…P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105841809 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚368235214944256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1627496361298800 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105831666 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7845165297268083e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12420791205888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192423822 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12491031281152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1639898391410549 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18666432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1637073688753664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚879 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1649494838423040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1639898749011183 _runtime‚9703.302451009 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1637073705793024 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12420791205888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86582011 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1637073705793024 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25538725 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12402387712383 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86582013 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1649494838423040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 192413677 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1637073688753664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86582013 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚355744183663104 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105831666 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105841809 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1649494479959552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStat¸7›µsMonitor.on_train_batch_start/inactive_split.all.freed‚44205154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86582011 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12421132630016 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12421132630016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12490917623296ï®Àÿ2›#- _wandbruntime‚9703  _runtime‚9703Šè³O'œ#ŠN C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44209417 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1649643864560640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1649644223031296 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12422242736128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192431126 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12421901304832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25541251 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1637221963255808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1649644223031296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192441285 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18668166 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚10699 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12492145566208 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1640046930466273 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚9703.930677966 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845165303549232e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86589806 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44209102 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚368269373042176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1627643774926912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86589806 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12403155539361 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86589804 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18668163 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192441285 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192431126 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚312 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896š³‡ù'#N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86589806 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44209102 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚368269373042176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1627643774926912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86589806 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12403155539361 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86589804 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18668163 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192441285 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192431126 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚312 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44209417 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1649643864560640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1649644223031296 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12422242736128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192431126 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12421901304832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25541251 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1637221963255808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1649644223031296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192441285 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18668166 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚10699 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12492145566208 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1640046930466273 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚9703.930677966 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845165303549232e+09ð2YTÃ2ž#- _wandbruntime‚9703  _runtime‚9703Úªº• Ÿ#ƒ 2loss/SRqqt1/assignment_loss‚0.5341212055654289 1loss/SRqqt1/detection_loss‚0.7765294658684169 %loss/total_loss‚7.797735335937482 epoch‚26 % _timestamp‚1.7845165303560023e+09 0loss/FRt1/assignment_loss‚1.5694633457196285 3loss/FBt1/assignment_loss‚0.0007890089002570301 1loss/SRqqt2/detection_loss‚0.5952420323503693 /loss/FBt2/detection_loss‚0.4630469730547206 _step‚881 0loss/FRt2/assignment_loss‚0.8443078568421627 2loss/SRqqt2/assignment_loss‚0.5327379494979231 1loss/FBt2/assignment_loss‚0.00655422910490609 /loss/FRt1/detection_loss‚0.4909904347668195 trainer/global_step‚10699 _runtime‚9703.935004632 .loss/FRt2/detection_loss‚1.549226133898537 .loss/FBt1/detection_loss‚0.434726700368312¥cÄÇ #ˆ 0loss/FRt2/assignment_loss‚0.8443078568421627 2loss/SRqqt2/assignment_loss‚0.5327379494979231 1loss/FBt2/assignment_loss‚0.00655422910490609 /loss/FRt1/detection_loss‚0.4909904347668195 trainer/global_step‚10699 _runtime‚9703.935004632 .loss/FRt2/detection_loss‚1.549226133898537 .loss/FBt1/detection_loss‚0.434726700368312 2loss/SRqqt1/assignment_loss‚0.5341212055654289 1loss/SRqqt1/detection_loss‚0.7765294658684169 %loss/total_loss‚7.797735335937482 epoch‚26 % _timestamp‚1.7845165303560023e+09 0loss/FRt1/assignment_loss‚1.5694633457196285 3loss/FBt1/assignment_loss‚0.0007890089002570301 1loss/SRqqt2/detection_loss‚0.5952420323503693 /loss/FBt2/detection_loss‚0.4630469730547206 _step‚881ñt¸x¡#‚(:í ³ŸöÒ»²«Ë disk.md0.in‚0" disk.nvme0c0n1.in‚ 166.93359375( disk.nvme0c0n1.out‚63240.90966796875& disk.nvme0n1.out‚63240.90966796875 network.recv‚ 2.558706e+06) disk./.usagePercent‚76.09157664751389 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 217.30859375% disk.nvme1c1n1.out‚31529.34765625 disk.nvme1n1.in‚ 217.30859375# disk.nvme1n1.out‚31529.34765625 disk.md0.out‚ 537.44140625# memory_percent‚97.0003604888916' proc.memory.availableMB‚ 491.4609375 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922 disk.nvme0n1.in‚ 166.93359375 network.sent‚ 1.7043247e+07$ disk./.usageGB‚1168.195930480957 disk.dm-0.in‚ 384.2421875 disk.rbd0.out‚0 cpu‚25.694418251653275 proc.cpu.threads‚17! disk.dm-0.out‚80741.880859375# disk.dm-1.out‚12913.24072265625²Jž¢#‚(:“ ³ŸöÒšœÌ gpu.0.gpu‚77.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.837' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.94566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CnäJu£#‚(:ê ŸöÒÃß´Ë disk.md0.out‚ 538.08984375 disk.nvme0c0n1.in‚ 171.046875% disk.nvme0c0n1.out‚63295.47265625# disk.nvme0n1.out‚63295.47265625) proc.memory.percent‚18.45989227294922% disk.nvme1n1.out‚31569.0556640625 disk.md0.in‚0# disk.dm-0.out‚80832.93505859375# disk.dm-1.out‚12915.08056640625 disk.rbd0.in‚0$ memory_percent‚96.99931144714355 proc.memory.rssMB‚ 3024.46875 proc.cpu.threads‚17 network.recv‚ 2.562926e+06) disk./.usagePercent‚76.09154360047032 disk.dm-0.in‚ 388.35546875 disk.rbd0.out‚0' proc.memory.availableMB‚ 491.6328125' disk.nvme1c1n1.out‚31569.0556640625 disk.nvme1n1.in‚ 217.30859375 disk.dm-1.in‚0 disk.nvme0n1.in‚ 171.046875 cpu‚25.693964248057398 network.sent‚ 1.71068e+07% disk./.usageGB‚1168.1954231262207" disk.nvme1c1n1.in‚ 217.30859375 ÷ø“¤#‚(:ˆ ŸöÒÂî€Ì gpu.0.gpu‚27.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.519' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.173 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fBþr¥#‚(:ç ÑŸöÒò–ªË# disk.dm-0.out‚80833.64599609375 disk.dm-1.in‚0$ memory_percent‚96.99935913085938) proc.memory.percent‚18.45989227294922 proc.cpu.threads‚17 disk.nvme1n1.in‚ 217.30859375 disk.dm-0.in‚ 388.35546875# disk.dm-1.out‚12915.26806640625 cpu‚25.693538138175843 network.sent‚ 1.7112466e+07$ disk./.usageGB‚1168.195369720459 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 171.046875 disk.nvme0n1.in‚ 171.046875" disk.nvme1c1n1.in‚ 217.30859375& disk.nvme1c1n1.out‚31569.947265625 disk.md0.out‚ 538.671875) disk./.usagePercent‚76.09154012183417 disk.rbd0.in‚0' disk.nvme0c0n1.out‚63296.7001953125$ disk.nvme1n1.out‚31569.947265625 disk.md0.in‚0# proc.memory.availableMB‚491.625 proc.memory.rssMB‚ 3024.46875 network.recv‚ 2.566455e+06% disk.nvme0n1.out‚63296.7001953125`!Èž¦#‚(:“ ÑŸöÒà´ëË gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.406' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.46866666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—O 2§#- _wandbruntime‚9747  _runtime‚9747 É|ɨ#à trainer/global_step‚10749 $ _timestamp‚1.784516573974055e+09 _runtime‚9747.550038054 _step‚882 ' lr-AdamW/pg1‚0.00028024308417941735 ' lr-AdamW/pg2‚0.00028024308417941735´ ˆ”Ω#È trainer/global_step‚10749 $ _timestamp‚1.784516573974055e+09 _runtime‚9747.550038054 _step‚882 ' lr-AdamW/pg1‚0.00028024308417941735 ' lr-AdamW/pg2‚0.00028024308417941735ò„cÞ2ª#- _wandbruntime‚9747  _runtime‚9747uñ©!«#ÿO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _step‚883 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86971663 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44406944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193298124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25653815 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1644487413860864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚357403197167104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1634867894672400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚9747.554097841 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86971661 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1644487430900224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25654072 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1656963774168576 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1644487413860864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106326461 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1644487430900224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106316318 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106316318 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369950023836672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚10749 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1656964132632064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86971661 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1647325407997557 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_»a´train_batch_start/allocated_bytes.all.allocated‚1656964132632064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369949908343808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1656963774168576 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1644487413860864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106326461 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1644487430900224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106316318 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106316318 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369950023836672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚10749 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1656964132632064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86971661 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1647325407997557 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1656964132632064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369949908343808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _step‚883 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86971663 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44406944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193298124 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25653815 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1644487413860864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚357403197167104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0óÏÏI(2­#- _wandbruntime‚9748  _runtime‚9748‹è—m'®#‰N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106325974 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86979454 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106336131 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1657113517240320 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12477811837952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193305428 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1635015325339872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚10749 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86979456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86979456 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1644635688363008 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1657113158769664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44411177 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357436358824960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12547940139520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86979454 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1644635705402368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18754863 % _timestamp‚1.7845165746015065e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193315587 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12547841168896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12477811837952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚9748.177308522 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44410885 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18754866 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369984317576704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚884 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193315587 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1644635688363008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106336131 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369984199993856 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1635015308300512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12458979314155 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25656022 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12477470406656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1647474304654027 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 239956131842ׇ‘ ¯#ŽN H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193305428 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1635015325339872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚10749 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86979456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86979456 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1644635688363008 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1657113158769664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44411177 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357436358824960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12547940139520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86979454 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1644635705402368 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18754863 % _timestamp‚1.7845165746015065e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193315587 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12547841168896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12477811837952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚9748.177308522 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44410885 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18754866 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369984317576704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚884 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193315587 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1644635688363008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106336131 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369984199993856 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1635015308300512 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12458979314155 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25656022 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ÍÁê­ƒBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12477470406656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1647474304654027 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106325974 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86979454 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106336131 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1657113517240320 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12477811837952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374ô0çºÎ2°#- _wandbruntime‚9748  _runtime‚97489¿ ±#… _step‚885 0loss/FRt2/assignment_loss‚0.7947419184558796 /loss/FRt2/detection_loss‚1.3345476216543857 1loss/SRqqt2/detection_loss‚0.6499535816607154 trainer/global_step‚10749 3loss/FBt1/assignment_loss‚0.0013796319080988729 /loss/FBt1/detection_loss‚0.6525228445997879 %loss/total_loss‚7.677363655461502 /loss/FRt1/assignment_loss‚1.717883418956029 3loss/SRqqt1/assignment_loss‚0.47228626856976036 % _timestamp‚1.7845165746030526e+09 2loss/SRqqt2/assignment_loss‚0.4845758210699698 2loss/FBt2/assignment_loss‚0.004748735870169493 /loss/FRt1/detection_loss‚0.4784558024185359 1loss/SRqqt1/detection_loss‚0.7222362614711089 /loss/FBt2/detection_loss‚0.3640317488270603 epoch‚26 _runtime‚ 9748.18205327î×%c²#Š 2loss/SRqqt2/assignment_loss‚0.4845758210699698 2loss/FBt2/assignment_loss‚0.004748735870169493 /loss/FRt1/detection_loss‚0.4784558024185359 1loss/SRqqt1/detection_loss‚0.7222362614711089 /loss/FBt2/detection_loss‚0.3640317488270603 epoch‚26 _runtime‚ 9748.18205327 _step‚885 0loss/FRt2/assignment_loss‚0.7947419184558796 /loss/FRt2/detection_loss‚1.3345476216543857 1loss/SRqqt2/detection_loss‚0.6499535816607154 trainer/global_step‚10749 3loss/FBt1/assignment_loss‚0.0013796319080988729 /loss/FBt1/detection_loss‚0.6525228445997879 %loss/total_loss‚7.677363655461502 /loss/FRt1/assignment_loss‚1.717883418956029 3loss/SRqqt1/assignment_loss‚0.47228626856976036 % _timestamp‚1.7845165746030526e+09õdSÎVv³#‚(:ë àŸöÒÕ©ËË" disk.nvme1c1n1.in‚ 217.30859375 network.sent‚ 1.7115489e+07 disk.dm-1.in‚0 disk.nvme1n1.in‚ 217.30859375& disk.nvme1n1.out‚31574.97119140625$ memory_percent‚96.99428081512451 proc.memory.rssMB‚ 3024.46875 cpu‚25.693111467799717 disk.md0.in‚0 disk.dm-0.in‚ 388.35546875 disk.nvme0c0n1.in‚ 171.046875( disk.nvme1c1n1.out‚31574.97119140625( proc.memory.availableMB‚ 492.45703125 network.recv‚ 2.569657e+06$ disk./.usageGB‚1166.861988067627 disk.dm-1.out‚12915.37109375 disk.nvme0n1.in‚ 171.046875) proc.memory.percent‚18.45989227294922 proc.cpu.threads‚17( disk.nvme0c0n1.out‚63314.46728515625) disk./.usagePercent‚76.00468901270978 disk.md0.out‚ 541.734375 disk.dm-0.out‚ 80850.1484375 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚63314.46728515625ðÛ|¨ž´#‚(:“ àŸöÒБ´Ì gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚273.193' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.06433333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç‰/wµ#‚(:ì ïŸöÒ°«ÛË proc.cpu.threads‚17" disk.nvme1c1n1.in‚ 217.30859375( disk.nvme1c1n1.out‚31575.66357421875 disk.nvme1n1.in‚ 217.30859375 disk.md0.in‚0 disk.md0.out‚ 542.19140625 disk.dm-0.in‚ 388.35546875 proc.memory.rssMB‚ 3024.46875 cpu‚25.692712399849867 network.sent‚ 1.7179979e+07 network.recv‚ 2.574274e+06% disk./.usageGB‚1166.8621215820312 disk.dm-1.in‚0 disk.dm-1.out‚12915.49609375 disk.rbd0.out‚0& disk.nvme0n1.out‚63315.62841796875& disk.nvme1n1.out‚31575.66357421875 disk.nvme0c0n1.in‚ 171.046875( disk.nvme0c0n1.out‚63315.62841796875$ memory_percent‚96.99299335479736) disk./.usagePercent‚76.00469770930019 disk.rbd0.in‚0 disk.nvme0n1.in‚ 171.046875 disk.dm-0.out‚ 80850.90625( proc.memory.availableMB‚ 492.66796875) proc.memory.percent‚18.45989227294922O™5Mž¶#‚(:“ ïŸöÒÛ‰¤Ì gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚273.239' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.07966666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢¬1+z·#‚(:ï þŸöÒúüÛË disk.dm-1.in‚0& disk.nvme0n1.out‚63317.95654296875$ memory_percent‚96.99299335479736( proc.memory.availableMB‚ 492.66796875% disk.nvme1n1.out‚31576.9912109375 disk.md0.out‚ 542.703125" disk.dm-0.out‚80853.0146484375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚63317.95654296875 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922 proc.cpu.threads‚17 network.sent‚ 1.7184807e+07 network.recv‚ 2.577389e+06 disk.nvme1n1.in‚ 217.30859375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 171.046875 cpu‚25.692289686289573) disk./.usagePercent‚76.00470019404031" disk.nvme1c1n1.in‚ 217.30859375' disk.nvme1c1n1.out‚31576.9912109375 disk.dm-0.in‚ 388.35546875# disk.dm-1.out‚12915.96337890625 disk.nvme0n1.in‚ 171.046875$ disk./.usageGB‚1166.862159729004 disk.md0.in‚0´G~œ¸#‚(:‘ þŸöÒ¨¨›Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.894' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.63133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºJ2¹#- _wandbruntime‚9792  _runtime‚9792Ë^•kʺ#Ä _step‚886 ' lr-AdamW/pg1‚0.00027779116373755926 ' lr-AdamW/pg2‚0.00027779116373755926 trainer/global_step‚10799 % _timestamp‚1.7845166185257497e+09 _runtime‚9792.101339553QùòÏ»#É ' lr-AdamW/pg1‚0.00027779116373755926 ' lr-AdamW/pg2‚0.00027779116373755926 trainer/global_step‚10799 % _timestamp‚1.7845166185257497e+09 _runtime‚9792.101339553 _step‚886öŸ…•2¼#- _wandbruntime‚9792  _runtime‚9792Tß(½#ÿO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚371664841710592 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1654752782185199 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1651901156007424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18839832 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1642239428046000 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106811113 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25769029 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚9792.106070422 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚887 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12531929409536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12602631039488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194172426 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87361313 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12513013577925 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12532270833664 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1664433068377600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87361311 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1664433068377600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359062210671104 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚10799 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚371664715731968 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12531929409536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 $ _timestamp‚1.784516618526853e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12532270833664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106811113 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25768740 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106811113 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25769029 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚9792.106070422 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚887 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12531929409536 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12602631039488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194172426 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87361313 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12513013577925 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12532270833664 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1664433068377600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87361311 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1664433068377600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359062210671104 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚10799 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚371664715731968 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12531929409536 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 $ _timestamp‚1.784516618526853e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12532270833664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106811113 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25768740 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1664582811449344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194189889 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1652049413470208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194179730 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194189889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1654900963640289 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _runtime‚9792.734659291 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12514121966177 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44612604 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194179730 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12533380939776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106820783 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1642386858713472 % _timestamp‚1.7845166191586747e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106810626 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106810626 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12603644011008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452€…¹'Á#N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1664582811449344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194189889 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1652049413470208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194179730 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚888 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194189889 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1654900963640289 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _runtime‚9792.734659291 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12514121966177 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44612604 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194179730 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12533380939776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106820783 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1642386858713472 % _timestamp‚1.7845166191586747e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106810626 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106810626 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12603644011008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106820783 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896ø˜Ü2Â#- _wandbruntime‚9792  _runtime‚9792ØDwƒ Ã#† 0loss/FRt1/assignment_loss‚1.7185242069640394 0loss/FRt2/assignment_loss‚0.8101145549049936 2loss/SRqqt2/assignment_loss‚0.5240619390112101 3loss/FBt1/assignment_loss‚0.0045199118639691165 /loss/FRt2/detection_loss‚1.4341461539247062 1loss/SRqqt1/detection_loss‚0.8185122124459866 /loss/FBt2/detection_loss‚0.5345509065976201 %loss/total_loss‚7.853652926165854 2loss/FBt2/assignment_loss‚0.008453063636277847 epoch‚26 trainer/global_step‚10799 $ _timestamp‚1.784516619159981e+09 _runtime‚ 9792.73904508 2loss/SRqqt1/assignment_loss‚0.4939496158389056 0loss/FRt1/detection_loss‚0.46391982906929535 _step‚889 1loss/SRqqt2/detection_loss‚0.6099029789629803 0loss/FBt1/detection_loss‚0.43299755294586945;ë-Ä#‹ 0loss/FRt1/assignment_loss‚1.7185242069640394 0loss/FRt2/assignment_loss‚0.8101145549049936 2loss/SRqqt2/assignment_loss‚0.5240619390112101 3loss/FBt1/assignment_loss‚0.0045199118639691165 /loss/FRt2/detection_loss‚1.4341461539247062 1loss/SRqqt1/detection_loss‚0.8185122124459866 /loss/FBt2/detection_loss‚0.5345509065976201 %loss/total_loss‚7.853652926165854 2loss/FBt2/assignment_loss‚0.008453063636277847 epoch‚26 trainer/global_step‚10799 $ _timestamp‚1.784516619159981e+09 _runtime‚ 9792.73904508 2loss/SRqqt1/assignment_loss‚0.4939496158389056 0loss/FRt1/detection_loss‚0.46391982906929535 _step‚889 1loss/SRqqt2/detection_loss‚0.6099029789629803 0loss/FBt1/detection_loss‚0.43299755294586945ùò{s0xÅ#‚(:í  öÒÉÛ¢Ë) disk./.usagePercent‚76.00468454017756 disk.nvme1n1.in‚ 217.30859375# disk.nvme1n1.out‚31577.47265625 disk.md0.out‚ 543.05078125" disk.dm-0.out‚80853.5888671875( disk.nvme0c0n1.out‚63318.84814453125" disk.nvme1c1n1.in‚ 217.30859375 disk.dm-1.in‚0# disk.dm-1.out‚12916.01025390625 disk.rbd0.in‚0$ memory_percent‚96.99475765228271 cpu‚25.69253325861296 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 171.046875 disk.nvme0n1.in‚ 171.046875& disk.nvme0n1.out‚63318.84814453125) proc.memory.percent‚18.45989227294922 network.sent‚ 1.7187771e+07 network.recv‚ 2.580802e+06% disk./.usageGB‚1166.8619194030762% disk.nvme1c1n1.out‚31577.47265625 disk.dm-0.in‚ 388.35546875( proc.memory.availableMB‚ 492.37890625 proc.memory.rssMB‚ 3024.46875 proc.cpu.threads‚17«—OžÆ#‚(:“  öÒ¦¸ÉË gpu.0.gpu‚97.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚270.607' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.20233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(ª>2Ç#- _wandbruntime‚9809  _runtime‚9809 4cò:7È# { 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7548564076423645 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5072087049484253 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.671678900718689 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24643829464912415 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6750895380973816 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/specificity‚0.9122923016548157 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04154467210173607 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9959622621536255 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_3‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.369252473115921 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.32649171352386475 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 trainer/global_step‚10799 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_2‚1 +particle/sensitivity‚0.6562320590019226 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.930077314376831 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚9809.825157496 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4761851131916046 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.16738903522491455 _step‚890 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6284427046775818 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03935207799077034 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6414343118667603 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3287450969219208 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3352409303188324 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8688237071037292 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7477847933769226 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6418641209602356 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7658724188804626 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 /particle/accuracy_4_of_4‚0.9503680467605591 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4904569089412689 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8396795392036438 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6173046827316284 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6975429058074951 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.056983739137649536 particle/accuracy_1_of_3‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2771814167499542 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5856218934059143 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6985557079315186 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2969580888748169 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3167317807674408 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16500000655651093 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24181608855724335 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7908541560173035 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5247153639793396 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04129200056195259 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04489494115114212 particle/accuracy_4_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6603419780731201 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3359951674938202 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19526426494121552 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7181974053382874 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2573452293872833 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6292140483856201 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1982409656047821 particle/accuracy_3_of_4‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7707294225692749 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7739863395690918 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7validation_average_jet_accuracy‚0.39825916290283203 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3241692781448364 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.040635380893945694 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7189502120018005 (particle/accuracy‚0.8573017120361328 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6284427046775818 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7845049500465393 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7835977673530579 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6805227398872375 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 +jet/accuracy_1_of_1‚0.36038708686828613 epoch‚26 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2915208041667938 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6671916842460632 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_1_of_3‚0.9307153820991516 )jet/accuracy_2_of_4‚0.788350522518158 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_1_of_4‚0.9763416051864624 particle/accuracy_2_of_4‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6638697981834412 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7539551258087158 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11967047303915024 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998782217502594 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7054741382598877 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2576815187931061 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6793397665023804 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.718938946723938 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7783230543136597 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2877192497253418 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31602758169174194 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22120626270771027 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 5 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.99656742811203 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6594271659851074 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6651217341423035 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7682350873947144 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚7.523996580947552 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9899393916130066 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10255104303359985 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.8036901354789734 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6713494658470154 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7246424555778503 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7210748791694641 +jet/accuracy_3_of_3‚0.07254147529602051 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4805780053138733 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03935207799077034 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6425366401672363 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_3_of_3‚0.9994120597839355 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6907975077629089 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_5_of_5‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4925577938556671 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.33619722723960876 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6997977495193481 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05214950442314148 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.909186840057373 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37978339195251465 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8789747357368469 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5533332824707031 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.615636944770813 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7495116591453552 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5793139338493347 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.28157979249954224 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6770833730697632 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7745490074157715 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06093251705169678 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4010416865348816 particle/accuracy_3_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.33459848165512085 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6261340379714966 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.32649171352386475 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04094276949763298 *jet/accuracy_2_of_3‚0.5965955257415771 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1iO#šlFRt*SRqqt*FBt/event_purity‚0.25552186369895935 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09832624346017838 % _timestamp‚1.7845166362491133e+09 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3749534487724304 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2087874710559845 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2710363268852234 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8396795392036438 jet/accuracy_1_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982032775878906 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.668214738368988 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18497875332832336 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_1‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37727904319763184 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941513538360596 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6984620690345764 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22544986009597778 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902462959289551 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794s­´#«=É#¥{ 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.33459848165512085 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6261340379714966 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.32649171352386475 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04094276949763298 *jet/accuracy_2_of_3‚0.5965955257415771 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.25552186369895935 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09832624346017838 % _timestamp‚1.7845166362491133e+09 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3749534487724304 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2087874710559845 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2710363268852234 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8396795392036438 jet/accuracy_1_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982032775878906 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.668214738368988 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18497875332832336 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_1‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37727904319763184 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941513538360596 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6984620690345764 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22544986009597778 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9902462959289551 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7548564076423645 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5072087049484253 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.671678900718689 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24643829464912415 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6750895380973816 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/specificity‚0.9122923016548157 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04154467210173607 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9959622621536255 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_3‚1 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.369252473115921 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.32649171352386475 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 trainer/global_step‚10799 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_2‚1 +particle/sensitivity‚0.6562320590019226 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.930077314376831 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚9809.825157496 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4761851131916046 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.16738903522491455 _step‚890 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6284427046775818 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03935207799077034 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6414343118667603 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3287450969219208 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3352409303188324 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8688237071037292 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7477847933769226 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6418641209602356 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7658724188804626 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 /particle/accuracy_4_of_4‚0.9503680467605591 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4904569089412689 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8396795392036438 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6173046827316284 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6975429058074951 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.056983739137649536 particle/accuracy_1_of_3‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2771814167499542 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5856218934059143 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6985557079315186 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2969580888748169 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3167317807674408 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16500000655651093 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24181608855724335 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7908541560173035 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5247153639793396 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04129200056195259 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04489494115114212 particle/accuracy_4_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6603419780731201 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3359951674938202 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19526426494121552 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7181974053382874 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2573452293872833 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6292140483856201 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1982409656047821 particle/accuracy_3_of_4‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7707294225692749 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7739863395690918 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7validation_average_jet_accuracy‚0.39825916290283203 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3241692781448364 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.040635380893945694 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7189502120018005 (particle/accuracy‚0.8573017120361328 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6284427046775818 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7845049500465393 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7835977673530579 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6805227398872375 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 +jet/accuracy_1_of_1‚0.36038708686828613 epoch‚26 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2915208041667938 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6671916842460632 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_1_of_3‚0.9307153820991516 )jet/accuracy_2_of_4‚0.788350522518158 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_1_of_4‚0.9763416051864624 particle/accuracy_2_of_4‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6638697981834412 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7539551258087158 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11967047303915024 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998782217502594 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7054741382598877 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2576815187931061 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6793397665023804 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.718938946723938 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7783230543136597 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2877192497253418 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31602758169174194 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.22120626270771027 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 5 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.99656742811203 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6594271659851074 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6651217341423035 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7682350873947144 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚7.523996580947552 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9899393916130066 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10255104303359985 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.8036901354789734 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6713494658470154 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7246424555778503 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7210748791694641 +jet/accuracy_3_of_3‚0.07254147529602051 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4805780053138733 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03935207799077034 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6425366401672363 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_3_of_3‚0.9994120597839355 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6907975077629089 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_5_of_5‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4925577938556671 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.33619722723960876 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6997977495193481 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05214950442314148 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.909186840057373 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37978339195251465 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8789747357368469 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5533332824707031 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.615636944770813 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7495116591453552 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5793139338493347 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.28157979249954224 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6770833730697632 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7745490074157715 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06093251705169678 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4010416865348816 particle/accuracy_3_of_5‚1ú+¡Ò'wÊ#‚(:ì œ öÒ˜ÖòË cpu‚25.697069927034452) disk./.usagePercent‚76.00470739978665' disk.nvme1c1n1.out‚31578.8408203125# disk.dm-0.out‚80858.69482421875 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922% disk./.usageGB‚1166.8622703552246" disk.nvme1c1n1.in‚ 217.30859375 disk.md0.in‚0 disk.md0.out‚ 543.3984375 disk.dm-0.in‚ 388.35546875# disk.dm-1.out‚12916.17041015625 disk.nvme0c0n1.in‚ 171.046875$ memory_percent‚96.99492454528809 network.sent‚ 1.7262089e+07 network.recv‚ 2.58397e+06% disk.nvme1n1.out‚31578.8408203125 disk.dm-1.in‚0& disk.nvme0c0n1.out‚63323.498046875' proc.memory.availableMB‚ 492.3515625 proc.cpu.threads‚17 disk.nvme1n1.in‚ 217.30859375 disk.nvme0n1.in‚ 171.046875$ disk.nvme0n1.out‚63323.498046875|›’Ë#‚(:‡ œ öÒÊî³Ì gpu.0.gpu‚18.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.031' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.677 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W€AQxÌ#‚(:í « öÒ×Ç·Ë disk.nvme0n1.in‚ 171.046875" disk.nvme1c1n1.in‚ 217.30859375 disk.nvme1n1.in‚ 217.30859375# disk.dm-1.out‚12916.25244140625% disk./.usageGB‚1166.8622131347656% disk.nvme1n1.out‚31579.2548828125# disk.dm-0.out‚80859.37451171875& disk.nvme0c0n1.out‚63324.306640625' proc.memory.availableMB‚ 492.6015625 cpu‚25.69730555606244 proc.cpu.threads‚17 disk.dm-0.in‚ 388.35546875$ disk.nvme0n1.out‚63324.306640625' disk.nvme1c1n1.out‚31579.2548828125 disk.md0.in‚0 disk.md0.out‚ 543.59765625 disk.rbd0.out‚0$ memory_percent‚96.99339866638184 network.sent‚ 1.7306517e+07 network.recv‚ 2.587998e+06 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 171.046875 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922) disk./.usagePercent‚76.00470367267647¤rµ¨žÍ#‚(:“ « öÒ ÄøË gpu.0.gpu‚81.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚274.915' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.63833333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•ZчwÎ#‚(:ì º öÒþžÏÌ% disk./.usageGB‚1166.8621215820312& disk.nvme1n1.out‚31579.87646484375" disk.dm-1.out‚12916.3818359375 disk.rbd0.in‚0 network.sent‚ 1.7313248e+07 network.recv‚ 2.591081e+06 disk.nvme1n1.in‚ 217.30859375 disk.md0.in‚0 disk.dm-0.in‚ 388.35546875 disk.dm-1.in‚0# disk.dm-0.out‚80860.72412109375 disk.nvme0c0n1.in‚ 171.046875$ disk.nvme0c0n1.out‚ 63325.8984375 disk.nvme0n1.in‚ 171.046875" disk.nvme0n1.out‚ 63325.8984375" disk.nvme1c1n1.in‚ 217.30859375( disk.nvme1c1n1.out‚31579.87646484375$ memory_percent‚96.99451923370361 disk.rbd0.out‚0) disk./.usagePercent‚76.00469770930019 disk.md0.out‚ 543.93359375( proc.memory.availableMB‚ 492.41796875 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922 cpu‚25.697151040967384 proc.cpu.threads‚17¹®´èÏ#‚(:„ º öÒïÌ¡Ì gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.1' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.7 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îß2Ð#- _wandbruntime‚9853  _runtime‚9853Ž\¾pÊÑ#Ä ' lr-AdamW/pg2‚0.00027534113504199947 trainer/global_step‚10849 % _timestamp‚1.7845166801930282e+09 _runtime‚9853.768691895 _step‚891 ' lr-AdamW/pg1‚0.00027534113504199947³»@†ÏÒ#É ' lr-AdamW/pg1‚0.00027534113504199947 ' lr-AdamW/pg2‚0.00027534113504199947 trainer/global_step‚10849 % _timestamp‚1.7845166801930282e+09 _runtime‚9853.768691895 _step‚891ûšúæ•2Ó#- _wandbruntime‚9853  _runtime‚9853ƒ?ö\(Ô#€P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1664121001028694 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1651532200388464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18930648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚892 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1661257207950848 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107377259 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107367116 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1661257224990208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195240885 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87873771 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1661257224990208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚10849 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1661257207950848 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1673865382761472 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12589141201504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1673865382761472 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1673865024297984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87873769 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12588800640230 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12607816347136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12607816347136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12608157771264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12679205670912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1651532217427824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44849506 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚360743802837504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107377259 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚9853.770869939 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25918561 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25918858 % _timestamp‚1.7845166801942413e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195251030 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87873771Î9Ä)_Õ#…P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25918561 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25918858 % _timestamp‚1.7845166801942413e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195251030 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87873771 ZLDeviceStatsMonitor.on_train_batch_start/i,^0¬%nactive_split_bytes.large_pool.peak‚ 626354176 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1664121001028694 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1651532200388464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18930648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚892 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1661257207950848 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107377259 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107367116 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1661257224990208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195240885 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87873771 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1661257224990208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚10849 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1661257207950848 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1673865382761472 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12589141201504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1673865382761472 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1673865024297984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87873769 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12588800640230 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12607816347136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12607816347136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12608157771264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12679205670912 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1651532217427824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44849506 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚360743802837504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107377259 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚9853.770869939 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274ü6Pm52Ö#- _wandbruntime‚9854  _runtime‚9854¶¬ ~'×#‰N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12608926446080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12590249589868 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195268493 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1661405499492352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚373457300179968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12680317072384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1661405482452992 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1664269540084418 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12609267877376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18932382 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚10849 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚360776964495360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44853257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1661405499492352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195268493 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109194240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44853529 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373457190985728 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12589909028482 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107386929 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25920878 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25921147 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚360776983107584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195258334 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87881564 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18932379 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12680226490368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1651679631055936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107376772 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1661405482452992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195258334 _step‚893 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87881564 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12609267877376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1674014767369728U‡7ë'Ø#ŽN C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25920878 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25921147 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚360776983107584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195258334 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87881564 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18932379 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12680226490368 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1651679631055936 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107376772 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1661405482452992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195258334 _step‚893 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87881564 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12609267877376 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1674014767369728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12608926446080 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12590249589868 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195268493 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1661405499492352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚373457300179968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12680317072384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1661405482452992 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1664269540084418 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12609267877376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18932382 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚10849 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚360776964495360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44853257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1661405499492352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195268493 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109194240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44853529 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚373457190985728 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12589909028482 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107386929 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647ýÓüàŽ2Ù#- _wandbruntime‚9854  _runtime‚9854Cg¦Õ Ú#„ $ _timestamp‚1.784516680824914e+09 _step‚894 /loss/FRt1/detection_loss‚0.5070327594096598 1loss/SRqqt2/detection_loss‚0.7138490821375565 /loss/FBt2/detection_loss‚0.4994914322804626 $loss/total_loss‚7.58841417865101 0loss/FRt1/assignment_loss‚1.5664189284658196 0loss/FRt2/assignment_loss‚0.7570207915006778 /loss/FRt2/detection_loss‚1.4691969509099436 /loss/FBt1/detection_loss‚0.3902740019300475 2loss/SRqqt1/assignment_loss‚0.5040489643459273 2loss/SRqqt2/assignment_loss‚0.4425746963044835 2loss/FBt1/assignment_loss‚0.016698712568204932 3loss/FBt2/assignment_loss‚0.0055865514538808725 trainer/global_step‚10849 _runtime‚9854.400380558 1loss/SRqqt1/detection_loss‚0.7162213073443462 epoch‚27>͘ÓÛ#‰ 2loss/SRqqt2/assignment_loss‚0.4425746963044835 2loss/FBt1/assignment_loss‚0.016698712568204932 3loss/FBt2/assignment_loss‚0.0055865514538808725 trainer/global_step‚10849 _runtime‚9854.400380558 1loss/SRqqt1/detection_loss‚0.7162213073443462 epoch‚27 $ _timestamp‚1.784516680824914e+09 _step‚894 /loss/FRt1/detection_loss‚0.5070327594096598 1loss/SRqqt2/detection_loss‚0.7138490821375565 /loss/FBt2/detection_loss‚0.4994914322804626 $loss/total_loss‚7.58841417865101 0loss/FRt1/assignment_loss‚1.5664189284658196 0loss/FRt2/assignment_loss‚0.7570207915006778 /loss/FRt2/detection_loss‚1.4691969509099436 /loss/FBt1/detection_loss‚0.3902740019300475 2loss/SRqqt1/assignment_loss‚0.5040489643459273þf5ã%zÜ#‚(:ï É öÒ®¦ËË( disk.nvme0c0n1.out‚63326.90576171875" disk.nvme1c1n1.in‚ 217.30859375 disk.nvme1n1.in‚ 217.30859375 disk.md0.in‚0 disk.md0.out‚ 544.24609375( proc.memory.availableMB‚ 491.65234375 network.recv‚ 2.594558e+06( disk./.usagePercent‚76.0047019333584 disk.dm-0.in‚ 388.35546875# disk.dm-0.out‚80861.45068359375 disk.nvme0c0n1.in‚ 171.046875& disk.nvme0n1.out‚63326.90576171875% disk.nvme1n1.out‚31580.3212890625) proc.memory.percent‚18.45989227294922" memory_percent‚96.999192237854 proc.memory.rssMB‚ 3024.46875 cpu‚25.696755738219935 proc.cpu.threads‚17 network.sent‚ 1.7329055e+07% disk./.usageGB‚1166.8621864318848" disk.dm-1.out‚12916.4208984375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 171.046875' disk.nvme1c1n1.out‚31580.3212890625 disk.dm-1.in‚0 disk.rbd0.out‚0ñû xžÝ#‚(:“ É öÒ¸ÿŒÌ gpu.0.gpu‚88.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚267.488' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.16266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rÍT VÞ#‚(:ó Ø öÒÁÁ©Ë disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 171.046875( disÛ'Q+(k.nvme0c0n1.out‚63329.39794921875& disk.nvme0n1.out‚63329.39794921875 disk.md0.out‚ 544.4140625 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922 proc.cpu.threads‚17 network.sent‚ 1.7380629e+07 disk.dm-0.in‚ 388.35546875 disk.nvme1n1.in‚ 217.30859375 disk.md0.in‚0$ memory_percent‚97.00009822845459( proc.memory.availableMB‚ 491.50390625 cpu‚25.696357492644005 network.recv‚ 2.598768e+06$ disk./.usageGB‚1166.862434387207 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 217.30859375( disk.nvme1c1n1.out‚31580.88427734375& disk.nvme1n1.out‚31580.88427734375) disk./.usagePercent‚76.00471808416917# disk.dm-0.out‚80863.97998046875# disk.dm-1.out‚12916.56005859375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 171.046875ê2ë„œß#‚(:‘ Ø öÒÒ®îË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚119.921' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.97366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!a1Koà#‚(:ä ç öÒŸ£žË network.sent‚ 1.7383652e+07) disk./.usagePercent‚76.00472106585731% disk./.usageGB‚1166.8624801635742 disk.dm-1.in‚0# disk.dm-1.out‚12916.65380859375 disk.nvme0c0n1.in‚ 171.046875% disk.nvme0c0n1.out‚63330.66796875 disk.md0.in‚0# disk.nvme0n1.out‚63330.66796875" disk.nvme1c1n1.in‚ 217.30859375$ disk.nvme1n1.out‚31581.798828125 disk.dm-0.in‚ 388.35546875# disk.dm-0.out‚80864.56201171875( proc.memory.availableMB‚ 491.71484375) proc.memory.percent‚18.45989227294922 cpu‚25.69595950236392 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 2.60224e+06 disk.nvme0n1.in‚ 171.046875& disk.nvme1c1n1.out‚31581.798828125 disk.nvme1n1.in‚ 217.30859375 disk.md0.out‚545.125$ memory_percent‚96.99881076812744 proc.memory.rssMB‚ 3024.46875Û_œá#‚(:‘ ç öÒñË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.049' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.34966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~·V2â#- _wandbruntime‚9898  _runtime‚9898òÎÂÒÊã#Ä % _timestamp‚1.7845167244810827e+09 _runtime‚9898.056705706 _step‚895 ' lr-AdamW/pg1‚0.00027289315545350284 ' lr-AdamW/pg2‚0.00027289315545350284 trainer/global_step‚10899`Ø0Ïä#É ' lr-AdamW/pg2‚0.00027289315545350284 trainer/global_step‚10899 % _timestamp‚1.7845167244810827e+09 _runtime‚9898.056705706 _step‚895 ' lr-AdamW/pg1‚0.00027289315545350284ÿ…v{L2å#- _wandbruntime‚9898  _runtime‚9898‚šøÚ(æ#þO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88263419 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375137778841600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1681334318507008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88263421 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19017348 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12663726873088 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚362402797729280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1681334676970496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272¥ß (ç#ƒP TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107861911 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12734981112320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1668670933058048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196125332 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1681334676970496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚362402797729280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1681334676970496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88263419 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375137778841600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1681334318507008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88263421 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19017348 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12663726873088 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚897 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45055572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19019082 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12736095330816 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12645392241890 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12736196398592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12664495547904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375172193010176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362435996611584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19019079 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196142795 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107871581 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26036490 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1659051164429536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1681484061578752 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1668819207560192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107871581 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12664836979200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196132636 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45055287 % _timestamp‚1.7845167251128287e+09 _runtime‚9898.688566269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196142795 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12664495547904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375172193010176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362435996611584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19019079 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196142795 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107871581 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26036490 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1659051164429536 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1681484061578752 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1668819207560192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107871581 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12664836979200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196132636 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45055287 % _timestamp‚1.7845167251128287e+09 _runtime‚9898.688566269 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196142795 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚897 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45055572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19019082 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12736095330816 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12645392241890 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12736196398592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452sõ®x2ë#- _wandbruntime‚9898  _runtime‚9898?Þ5„ ì#ƒ 1loss/SRqqt2/detection_loss‚0.7105046717101872 /loss/FBt1/detection_loss‚0.4749234268120268 %loss/total_loss‚8.284414716948085 /loss/FRt1/assignment_loss‚1.689796179811531 2loss/SRqqt2/assignment_loss‚0.5439679927211754 /loss/FRt1/detection_loss‚0.5046948574314619 1loss/SRqqt1/detection_loss‚0.6785402605895144 epoch‚27 _runtime‚9898.693010828 0loss/FRt2/assignment_loss‚0.8735304497294689 /loss/FBt2/detection_loss‚0.6708591545807551 $ _timestamp‚1.784516725113888e+09 2loss/FBt2/assignment_loss‚0.019511244794086897 /loss/FRt2/detection_loss‚1.5400340261906573 trainer/global_step‚10899 _step‚898 2loss/SRqqt1/assignment_loss‚0.5657394903163853 2loss/FBt1/assignment_loss‚0.012312962260834198Šž-í#ˆ 2loss/FBt2/assignment_loss‚0.019511244794086897 /loss/FRt2/detection_loss‚1.5400340261906573 trainer/global_step‚10899 _step‚898 2loss/SRqqt1/assignment_loss‚0.5657394903163853 2loss/FBt1/assignment_loss‚0.012312962260834198 1loss/SRqqt2/detection_loss‚0.7105046717101872 /loss/FBt1/detection_loss‚0.4749234268120268 %loss/total_loss‚8.284414716948085 /loss/FRt1/assignment_loss‚1.689796179811531 2loss/SRqqt2/assignment_loss‚0.5439679927211754 /loss/FRt1/detection_loss‚0.5046948574314619 1loss/SRqqt1/detection_loss‚0.6785402605895144 epoch‚27 _runtime‚9898.693010828 0loss/FRt2/assignment_loss‚0.8735304497294689 /loss/FBt2/detection_loss‚0.6708591545807551 $ _timestamp‚1.784516725113888e+09‚Åš¬:{î#‚(:ð ö öÒ£÷ÇË disk.md0.in‚0# disk.dm-0.out‚80866.30419921875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 171.046875 disk.nvme0n1.in‚ 171.046875& disk.nvme0n1.out‚63332.64208984375" memory_percent‚97.001051902771 proc.memory.rssMB‚ 3024.46875 network.sent‚ 1.7389626e+07 network.recv‚ 2.605676e+06 disk.dm-0.in‚ 388.35546875 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚1166.862491607666 disk.nvme1n1.in‚ 217.30859375 disk.md0.out‚ 545.5078125# disk.dm-1.out‚12916.74267578125( disk.nvme0c0n1.out‚63332.64208984375( disk.nvme1c1n1.out‚31582.50244140625( proc.memory.availableMB‚ 491.34765625) disk./.usagePercent‚76.00472181127934& disk.nvme1n1.out‚31582.50244140625" disk.nvme1c1n1.in‚ 217.30859375) proc.memory.percent‚18.45989227294922 cpu‚25.69556046857193Õ«ûžï#‚(:“ ö öÒõØ°Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.559' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.85300000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÆñDtð#‚(:é …¡öÒʺºË( proc.memory.availableMB‚ 491.58203125 cpu‚25.695164959936577 network.sent‚ 1.7452036e+07 network.recv‚ 2.609805e+06# disk.dm-1.out‚12916.83251953125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚63333.4951171875 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922# disk.nvme1c1n1.out‚ 31583.015625 disk.nvme1n1.in‚ 217.30859375 disk.md0.out‚ 545.83203125 disk.dm-0.in‚ 388.35546875# disk.dm-0.out‚80866.86669921875 disk.dm-1.in‚0% disk.nvme0n1.out‚63333.4951171875$ memory_percent‚96.99962139129639) disk./.usagePercent‚76.00472131433132% disk./.usageGB‚1166.8624839782715" disk.nvme1c1n1.in‚ 217.30859375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 171.046875 disk.nvme0n1.in‚ 171.046875 proc.cpu.threads‚17! disk.nvme1n1.out‚ 31583.015625 disk.md0.in‚0Á3;Õžñ#‚(:“ …¡öÒÂÙˆÌ gpu.0.gpu‚55.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.442' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.14733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<Ö}uò#‚(:ê ”¡öÒ”ݸË cpu‚25.69484557268827 proc.cpu.threads‚17 disk.dm-0.in‚ 388.35546875 network.sent‚ 1.7457814e+07 network.recv‚ 2.613153e+06 disk.md0.out‚ 546.42578125 disk.dm-1.in‚0# memory_percent‚96.9996690750122 proc.memory.rssMB‚ 3024.46875 disk.nvme0n1.in‚ 171.046875" disk.nvme0n1.out‚ 63335.8984375" disk.dm-1.out‚12916.9404296875( proc.memory.availableMB‚ 491.57421875) proc.memory.percent‚18.45989227294922% disk./.usageGB‚1166.8625984191895" disk.nvme1c1n1.in‚ 217.30859375( disk.nvme1c1n1.out‚31583.98583984375 disk.nvme1n1.in‚ 217.30859375 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 63335.8984375) disk./.usagePercent‚76.00472876855167& disk.nvme1n1.out‚31583.98583984375# disk.dm-0.out‚80868.86865234375 disk.nvme0c0n1.in‚ 171.046875M¢XZœó#‚(:‘ ”¡öÒŠ·øË gpu.0.gpu‚4.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.649' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.21633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"2Éç2ô#- _wandbruntime‚9942  _runtime‚9942EÇ­(Éõ#à % _timestamp‚1.7845167689495733e+09 _runtime‚ 9942.52521889 _step‚899 ' lr-AdamW/pg1‚0.00027044738220122373 ' lr-AdamW/pg2‚0.00027044738220122373 trainer/global_step‚10949Cr½SÎö#È % _timestamp‚1.7845167689495733e+09 _runtime‚ 9942.52521889 _step‚899 ' lr-AdamW/pg1‚0.00027044738220122373 ' lr-AdamW/pg2‚0.00027044738220122373 trainer/global_step‚10949ƒVEP­2÷#- _wandbruntime‚9942  _runtime‚9942iDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88653071 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1688803971179520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26149447 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12700107628320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1688803971179520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26149159 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12719295974912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364061811233280 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12718954550784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19104048 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88653069 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1676084658165248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1676084675204608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12790964347904 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1688803612716032 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1678975034202710 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1688803612716032 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12699767067046 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚9942.530120839 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108346563 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108336420 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10949 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1678975391803344 _step‚900 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196999634 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88653071 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88653069 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364061829845504 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108346563 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12719295974912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45253495 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232(0Eù#…P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1688803971179520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26149447 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12700107628320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1688803971179520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26149159 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12719295974912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364061811233280 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12718954550784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19104048 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88653069 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1676084658165248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceSÖKXŒtatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1676084675204608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12790964347904 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1688803612716032 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1678975034202710 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1688803612716032 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12699767067046 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _runtime‚9942.530120839 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108346563 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108336420 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚10949 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1678975391803344 _step‚900 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196999634 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88653071 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88653069 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364061829845504 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108346563 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12719295974912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45253495 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88653071„º|Á2ú#- _wandbruntime‚9943  _runtime‚9943Íó'û#ŠN REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88660864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45257489 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26151707 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1679123930859180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 197006938 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚9943.154980104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108346076 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88660864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1666422697803136 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _step‚901 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1688952997317120 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1688953355787776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12792077592576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 % _timestamp‚1.7845167695791888e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12720406081024 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1676232949706752 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12791968136192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197017097 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108356233 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26151426 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364095010115584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88660862 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12720064649728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108346076 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197006938 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1676232932667392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1688953355787776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19105779 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1666422714842496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45257205 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12701216016684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376887087708160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1676232932667392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19105782 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197017097 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12720064649728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364094991503360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚10949 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88660864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1666422697803136 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _step‚901 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1688952997317120 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1688953355787776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12792077592576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 % _timestamp‚1.7845167695791888e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12720406081024 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1676232949706752 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12791968136192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197017097 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108356233 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26151426 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364095010115584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88660862 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12720064649728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108346076 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197006938 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1676232932667392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1688953355787776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19105779 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1666422714842496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45257205 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12701216016684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚376887087708160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1676232932667392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19105782 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197017097 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12720064649728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364094991503360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚10949 QDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1696272906925056 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26264546 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845168136110487e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378567674684928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12846730260480 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12774865076736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12755250280454 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89042719 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1683498400311808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1673646817548624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197863791 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12774865076736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108831215 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108821072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26264264 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚10999 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89042721 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45455009 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _step‚904 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1696273265388544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1696272906925056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1683498383272448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1683498400311808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19190748 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1696272906925056 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26264546 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845168136110487e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378567674684928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12846730260480 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12774865076736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12755250280454 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89042719 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1683498400311808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1673646817548624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197863791 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12774865076736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108831215 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108821072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26264264 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚10999 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89042721 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45455009 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _step‚904 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1696273265388544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1696272906925056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1683498383272448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1683498400311808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19190748 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197891399 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚9987.819048636 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1696422649996800 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1683646657774592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12775975182848 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1683646674813952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12756358668706 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89050514 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1696422291526144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1686550947446188 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108830728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19192479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378601971847168 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1673794248216096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89050512 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1686550589845442 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197891399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19192482 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12775633751552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108840885 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 197881240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12847948227584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89050514 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45459016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1696422649996800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378601856361472 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚10999 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1683646674813952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26266537 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108830728 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12756699230092 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45459302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚365754005007360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12775633751552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12775975182848 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286]\¥u©Ž$ŒN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197891399 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19192482 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12775633751552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108840885 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 197881240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12847948227584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89050514 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45459016 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1696422649996800 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378601856361472 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚10999 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1683646674813952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26266537 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108830728 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12756699230092 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45459302 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚365754005007360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12775633751552 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12775975182848 JBDeviceStatsMonitor.on_train_batch_end/segmenÑÅnÖit.small_pool.allocated‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1683646657774592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26266820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197891399 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚9987.819048636 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1696422649996800 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1683646657774592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12775975182848 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1683646674813952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12756358668706 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89050514 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1696422291526144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1686550947446188 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108830728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19192479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378601971847168 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1673794248216096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89050512 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1686550589845442 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344‰üÑ!2$- _wandbruntime‚9987  _runtime‚9987Ë}ø $… trainer/global_step‚10999 _runtime‚9987.819776727 _step‚906 0loss/FRt2/assignment_loss‚0.7339536969794519 2loss/SRqqt1/assignment_loss‚0.5241293729899886 2loss/SRqqt2/assignment_loss‚0.5407704318389946 /loss/FRt1/detection_loss‚0.5135983415318099 .loss/FRt2/detection_loss‚1.674984276440459 $loss/total_loss‚8.03463025690439 % _timestamp‚1.7845168142444081e+09 3loss/FBt1/assignment_loss‚0.0029181543760799173 /loss/FBt1/detection_loss‚0.4598947983129971 /loss/FBt2/detection_loss‚0.6695689853569944 0loss/FRt1/assignment_loss‚1.5419306984487569 3loss/FBt2/assignment_loss‚0.0005228751675965045 1loss/SRqqt1/detection_loss‚0.7026146272364618 1loss/SRqqt2/detection_loss‚0.6697439982247965 epoch‚27î6j‘$Š 0loss/FRt1/assignment_loss‚1.5419306984487569 3loss/FBt2/assignment_loss‚0.0005228751675965045 1loss/SRqqt1/detection_loss‚0.7026146272364618 1loss/SRqqt2/detection_loss‚0.6697439982247965 epoch‚27 trainer/global_step‚10999 _runtime‚9987.819776727 _step‚906 0loss/FRt2/assignment_loss‚0.7339536969794519 2loss/SRqqt1/assignment_loss‚0.5241293729899886 2loss/SRqqt2/assignment_loss‚0.5407704318389946 /loss/FRt1/detection_loss‚0.5135983415318099 .loss/FRt2/detection_loss‚1.674984276440459 $loss/total_loss‚8.03463025690439 % _timestamp‚1.7845168142444081e+09 3loss/FBt1/assignment_loss‚0.0029181543760799173 /loss/FBt1/detection_loss‚0.4598947983129971 /loss/FBt2/detection_loss‚0.6695689853569944Šoñ+o’$‚(:ä СöÒº´Ë disk.dm-0.in‚ 388.35546875 disk.dm-1.in‚0 disk.nvme0n1.in‚ 171.046875$ memory_percent‚97.00191020965576 proc.memory.rssMB‚ 3024.46875 network.sent‚ 1.7533115e+07 disk.rbd0.out‚0( proc.memory.availableMB‚ 491.20703125) proc.memory.percent‚18.45989227294922" disk.nvme1c1n1.in‚ 217.30859375" disk.nvme1c1n1.out‚ 31586.59375 disk.nvme1n1.in‚ 217.30859375# disk.dm-0.out‚80874.61865234375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 171.046875 network.recv‚ 2.628204e+06' disk./.usagePercent‚76.004714605533% disk./.usageGB‚1166.8623809814453# disk.dm-1.out‚12917.30712890625' disk.nvme0c0n1.out‚63342.4521484375% disk.nvme0n1.out‚63342.4521484375 cpu‚25.69345255205677 proc.cpu.threads‚17 disk.nvme1n1.out‚ 31586.59375 disk.md0.in‚0 disk.md0.out‚ 547.83203125­º¯¾“$‚(:’ СöÒà¼ôË gpu.0.gpu‚20.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚285.509' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.16966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØYËr”$‚(:ç ß¡öÒ¦‰§Ë disk.md0.out‚ 548.24609375 disk.dm-0.in‚ 388.35546875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 171.046875 disk.nvme0n1.out‚ 63344.03125 disk.md0.in‚0# disk.dm-0.out‚80876.13427734375 disk.rbd0.out‚0 disk.nvme1n1.in‚ 217.30859375% disk.nvme1n1.out‚31587.5478515625) proc.memory.percent‚18.45989227294922 proc.cpu.threads‚17" disk.nvme0c0n1.out‚ 63344.03125) disk./.usagePercent‚76.00476355491331# disk.dm-1.out‚12917.43212890625 disk.nvme0n1.in‚ 171.046875" disk.nvme1c1n1.in‚ 217.30859375$ memory_percent‚97.00191020965576( proc.memory.availableMB‚ 491.20703125 proc.memory.rssMB‚ 3024.46875 network.recv‚ 2.632327e+06 disk.dm-1.in‚0' disk.nvme1c1n1.out‚31587.5478515625 cpu‚25.693113644013494 network.sent‚ 1.7595637e+07% disk./.usageGB‚1166.8631324768066ÚɃž•$‚(:“ ß¡öÒ–‡¹Ë gpu.0.gpu‚41.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.497' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.16566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BhÊy–$‚(:î î¡öÒ܉ÄË disk.nvme1n1.in‚ 217.30859375$ memory_percent‚97.00191020965576) proc.memory.percent‚18.45989227294922 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 171.38671875 network.sent‚ 1.7600501e+07 network.recv‚ 2.635664e+06 disk.md0.in‚0 disk.md0.out‚ 550.53515625 disk.dm-0.in‚ 388.6953125" disk.nvme1c1n1.in‚ 217.30859375# disk.nvme1n1.out‚31596.35546875 disk.rbd0.in‚0% disk.nvme1c1n1.out‚31596.35546875 proc.memory.rssMB‚ 3024.46875 cpu‚25.69274908807455) disk./.usagePercent‚76.09171554448648 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 171.38671875& disk.nvme0n1.out‚64089.88623046875( proc.memory.availableMB‚ 491.20703125% disk./.usageGB‚1168.1980628967285 disk.dm-0.out‚81622.30078125# disk.dm-1.out‚12920.88720703125( disk.nvme0c0n1.out‚64089.88623046875 ¤¦¸“—$‚(:ˆ î¡öÒ„×Ë gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.183' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.061 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†‰Ó4˜$/ _wandbruntime‚10031  _runtime‚10031åeÕ É™$à & lr-AdamW/pg1‚0.0002655630829033034 & lr-AdamW/pg2‚0.0002655630829033034 trainer/global_step‚11049 % _timestamp‚1.7845168580799236e+09 _runtime‚10031.655518561 _step‚907 ¥ +Κ$È % _timestamp‚1.7845168580799236e+09 _runtime‚10031.655518561 _step‚907 & lr-AdamW/pg1‚0.0002655630829033034 & lr-AdamW/pg2‚0.0002655630829033034 trainer/global_step‚11049‹r›á‚4›$/ _wandbruntime‚10031  _runtime‚10031@œá(œ$P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198748238 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _step‚908 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12830092754432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380282446284288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12830434178560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1693829424977360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89432369 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45656791 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19277445 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12902713312256 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109315867 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12902608043008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1703742559597568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45657071 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1690912108379648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12830092754432 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109305724 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367379856853504 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109315867 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1703742201134080 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12830434178560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1681018350922224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89432371 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1703742201134080 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89432369 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198738093 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26379623 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380282570165760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1690912125419008 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1690912108379648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19277448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12830092754432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚380282446284288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12830434178560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1693829424977360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89432369 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45656791 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19277445 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12902713312256 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109315867 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12902608043008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1703742559597568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45657071 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1690912108379648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12830092754432 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109305724 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367379856853504 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109315867 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1703742201134080 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12830434178560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1681018350922224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89432371 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1703742201134080 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89432369 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198738093 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26379623 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚380282570165760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1690912125419008 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1690912108379648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19277448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380316862897152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1691060382881792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89440164 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109325537 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109315380 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89440162 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367413037123584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12903825773568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845168587129977e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198755542 trainer/global_step‚11049 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198765701 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26381896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12812182443500 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198765701 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19279179 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12831202853376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45660797 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12831544284672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12903724705792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0èL_^' $N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12831202853376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45660797 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12831544284672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12903724705792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198755542 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109325537 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚380316862897152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1691060382881792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89440164 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109325537 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109315380 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89440162 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367413037123584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12903825773568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845168587129977e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198755542 trainer/global_step‚11049 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198765701 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26381896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12812182443500 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198765701 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19279179 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85œ(¤ö4¡$/ _wandbruntime‚10032  _runtime‚10032µñ»©¢$‚ /loss/FRt1/assignment_loss‚1.625523800967876 0loss/FRt2/assignment_loss‚0.8123786959635175 .loss/FRt2/detection_loss‚1.617814356782575 0loss/SRqqt2/detection_loss‚0.709271799105756 trainer/global_step‚11049 _step‚910 /loss/FRt1/detection_loss‚0.5087382842047149 % _timestamp‚1.7845168587141151e+09 _runtime‚10032.289496224 2loss/SRqqt1/assignment_loss‚0.5399680844014363 2loss/FBt1/assignment_loss‚0.003562311249886532 1loss/FBt2/assignment_loss‚0.02563146977594588 /loss/FBt1/detection_loss‚0.3116339544465628 epoch‚27 2loss/SRqqt2/assignment_loss‚0.4960775522112011 1loss/SRqqt1/detection_loss‚0.6382674651642289 /loss/FBt2/detection_loss‚0.9165049675308788 %loss/total_loss‚8.205372741804581¬°e‹ £$‡ .loss/FRt2/detection_loss‚1.617814356782575 0loss/SRqqt2/detection_loss‚0.709271799105756 trainer/global_step‚11049 _step‚910 /loss/FRt1/detection_loss‚0.5087382842047149 % _timestamp‚1.7845168587141151e+09 _runtime‚10032.289496224 2loss/SRqqt1/assignment_loss‚0.5399680844014363 2loss/FBt1/assignment_loss‚0.003562311249886532 1loss/FBt2/assignment_loss‚0.02563146977594588 /loss/FBt1/detection_loss‚0.3116339544465628 epoch‚27 2loss/SRqqt2/assignment_loss‚0.4960775522112011 1loss/SRqqt1/detection_loss‚0.6382674651642289 /loss/FBt2/detection_loss‚0.9165049675308788 %loss/total_loss‚8.205372741804581 /loss/FRt1/assignment_loss‚1.625523800967876 0loss/FRt2/assignment_loss‚0.8123786959635175ŽY—ÖŠy¤$‚(:î ý¡öÒÚ¢êË# disk.nvme0n1.out‚64718.41796875' disk.nvme1c1n1.out‚31597.4248046875# disk.dm-0.out‚82250.48974609375 disk.dm-1.in‚0$ memory_percent‚97.00515270233154 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922) disk./.usagePercent‚76.09167554017058 disk.rbd0.in‚0 disk.nvme0n1.in‚ 175.52734375 disk.nvme1n1.in‚ 217.30859375 disk.md0.in‚0 disk.md0.out‚ 551.34765625# disk.dm-1.out‚12920.98095703125 cpu‚25.69238520560643 network.sent‚ 1.761817e+07 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 217.30859375% disk.nvme1n1.out‚31597.4248046875 proc.cpu.threads‚17 network.recv‚ 2.639239e+06% disk./.usageGB‚1168.1974487304688" disk.nvme0c0n1.in‚ 175.52734375 disk.dm-0.in‚ 392.8359375( proc.memory.availableMB‚ 490.67578125% disk.nvme0c0n1.out‚64718.41796875[åd“¥$‚(:ˆ ý¡öÒ«¥¯Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.969' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.323 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù™Ìàx¦$‚(:í Œ¢öҠؘË% disk./.usageGB‚1168.1974143981934 disk.dm-0.in‚ 392.8359375# disk.dm-0.out‚82253.37841796875( disk.nvme1c1n1.out‚31598.96337890625 disk.md0.in‚0 disk.md0.out‚ 551.90625 cpu‚25.692052267951432 network.sent‚ 1.7667837e+07 network.recv‚ 2.643052e+06 disk.dm-1.out‚12921.40234375 disk.rbd0.in‚0& disk.nvme0c0n1.out‚64721.384765625 disk.nvme0n1.in‚ 175.52734375" disk.nvme1c1n1.in‚ 217.30859375& disk.nvme1n1.out‚31598.96337890625) disk./.usagePercent‚76.09167330390447 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 175.52734375$ disk.nvme0n1.out‚64721.384765625$ memory_percent‚97.00207710266113' proc.memory.availableMB‚ 491.1796875) proc.memory.percent‚18.45989227294922 disk.dm-1.in‚0 disk.nvme1n1.in‚ 217.30859375 proc.memory.rssMB‚ 3024.46875 proc.cpu.threads‚17/¶À“§$‚(:ˆ Œ¢öÒ¾åÕË gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚276.996' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü¢7Ð{¨$‚(:𠛢öÒØê§Ë( proc.memory.availableMB‚ 491.17578125) proc.memory.percent‚18.45989227294922 network.recv‚ 2.646542e+06 disk.md0.out‚ 552.19140625 disk.dm-0.in‚ 392.8359375" disk.nvme0c0n1.in‚ 175.52734375 proc.memory.rssMB‚ 3024.46875 cpu‚25.69171628086815% disk./.usageGB‚1168.1968154907227 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚64729.416015625 proc.cpu.threads‚17# disk.dm-0.out‚82261.16357421875 network.sent‚ 1.7672659e+07) disk./.usagePercent‚76.09163429348462" disk.nvme1c1n1.in‚ 217.30859375 disk.nvme1n1.in‚ 217.30859375& disk.nvme1n1.out‚31599.39697265625& disk.nvme0c0n1.out‚64729.416015625 disk.nvme0n1.in‚ 175.52734375( disk.nvme1c1n1.out‚31599.39697265625 disk.dm-1.out‚12921.44140625$ memory_percent‚97.00210094451904¢åž©$‚(:“ ›¢öÒ­ßýË gpu.0.gpu‚39.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.985' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.66166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/¹4ª$/ _wandbruntime‚10076  _runtime‚10076Хɫ$à _runtime‚ 10076.3102669 _step‚911 ' lr-AdamW/pg1‚0.00026312487056707876 ' lr-AdamW/pg2‚0.00026312487056707876 trainer/global_step‚11099 % _timestamp‚1.7845169027346778e+09Ъ¬²Î¬$È ' lr-AdamW/pg1‚0.00026312487056707876 ' lr-AdamW/pg2‚0.00026312487056707876 trainer/global_step‚11099 % _timestamp‚1.7845169027346778e+09 _runtime‚ 10076.3102669 _step‚911)ÿ¦¢4­$/ _wandbruntime‚10076  _runtime‚10076Ê!@O®$P ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1711211853806592 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1698325833486848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1688389867256464 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1711211495343104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1711211853806592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12885661856256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45859970 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚381997449671680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12866557268544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12885661856256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19364148 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109790376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚369038870357504 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199622540 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1688389884295824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199612395 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26495541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12958471947776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109800519 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8982ÈË;j¸ 2019 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12886003280384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚11099 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109800519 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109790376 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45859686 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚381997323693056 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1701256083963734 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19364145 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1701256441564368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845169027357383e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12958579314176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1711211495343104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1711211853806592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12885661856256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45859970 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚381997449671680 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12866557268544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12885661856256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19364148 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109790376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚369038870357504 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199622540 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1688389884295824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199612395 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26495541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12958471947776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109800519 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89822019 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12886003280384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0RL74°$/ _wandbruntime‚10076  _runtime‚10076õG1”'±$ŠN L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1698474107988992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26497868 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12959691857408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89829812 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45864030 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199640003 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1688537297923936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚369072032015360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1688537314963296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89829814 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1698474107988992 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1698474125028352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1701404623019458 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12867325095522 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89829812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45863747 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚382031624902144 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45863747 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚382031624902144 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1698474107988992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26497868 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12959691857408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89829812 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45864030 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199640003 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1688537297923936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚369072032015360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1688537314963296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89829814 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1698474107988992 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1698474125028352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1701404623019458 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12867325095522 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89829812 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12941230958080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12941230958080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1705739558594048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46061696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12922040481952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110285171 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚370697883861504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110275028 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200496842 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1708683100550742 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110285171 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1718681148015616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90211669 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90211671 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚370697865249280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1718681148015616 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1718680789552128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1705739575633408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12941572382208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1718680789552128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚11149 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200486697 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90211669 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19450845 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90211671 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26610848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19450848 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12941572382208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46061397 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚916 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1708683458151376 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1695761400630064 _runtime‚10120.605918721 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12941230958080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12941230958080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1705739558594048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46061696 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12922040481952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110285171 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚370697883861504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110275028 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200496842 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1708683100550742 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110285171 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1718681148015616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90211669 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90211671 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚370697865249280 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1718681148015616 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1718680789552128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1705739575633408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12941572382208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1718680789552128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚11149 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200486697 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90211669 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19450845 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90211671 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26610848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19450848 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12941572382208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46061397 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚916 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1708683458151376 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1695761400630064 _runtime‚10120.605918721 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1695908831297536 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200504146 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110284684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚383746622731264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1718830532623872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_8Çý©pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370731045519360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110284684 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚10121.237439821 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1705887850135552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370731064131584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1718830174153216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26613168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12942341057024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200514305 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90219462 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46065449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19452582 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1708831997207212 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200514305 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3¯—„'Ä$N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370731045519360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110284684 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚10121.237439821 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1705887850135552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370731064131584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1718830174153216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26613168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12942341057024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200514305 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90219462 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚298 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46065449 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19452582 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1708831997207212 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200514305 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚917 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12942341057024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12942682488320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1705887833096192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200504146 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12923148870316 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90219464 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110294841 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26612870 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12922808308930 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1705887850135552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19452579 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1708831639606466 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1695908831297536 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200504146 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚301 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110284684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚383746622731264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1718830532623872•sèb4Å$/ _wandbruntime‚10121  _runtime‚10121ÖvîÆ$‰ _step‚918 3loss/FBt1/assignment_loss‚0.0009600766812926343 /loss/FRt2/detection_loss‚1.4699909941406875 0loss/FBt1/detection_loss‚0.36023749565455065 _runtime‚10121.238064561 0loss/FRt1/assignment_loss‚1.6610676701965295 2loss/SRqqt1/assignment_loss‚0.4761501596999383 1loss/SRqqt1/detection_loss‚0.6686894770770586 1loss/SRqqt2/detection_loss‚0.6017830208600936 3loss/FBt2/assignment_loss‚0.0011194485169002922 /loss/FRt1/detection_loss‚0.4301699953931369 /loss/FBt2/detection_loss‚0.4607952482005881 %loss/total_loss‚7.392170153858848 epoch‚27 0loss/FRt2/assignment_loss‚0.7524094170186578 2loss/SRqqt2/assignment_loss‚0.5087971504194145 trainer/global_step‚11149 % _timestamp‚1.7845169476627738e+09·ÌROÇ$Ž 0loss/FRt1/assignment_loss‚1.6610676701965295 2loss/SRqqt1/assignment_loss‚0.4761501596999383 1loss/SRqqt1/detection_loss‚0.6686894770770586 1loss/SRqqt2/detection_loss‚0.6017830208600936 3loss/FBt2/assignment_loss‚0.0011194485169002922 /loss/FRt1/detection_loss‚0.4301699953931369 /loss/FBt2/detection_loss‚0.4607952482005881 %loss/total_loss‚7.392170153858848 epoch‚27 0loss/FRt2/assignment_loss‚0.7524094170186578 2loss/SRqqt2/assignment_loss‚0.5087971504194145 trainer/global_step‚11149 % _timestamp‚1.7845169476627738e+09 _step‚918 3loss/FBt1/assignment_loss‚0.0009600766812926343 /loss/FRt2/detection_loss‚1.4699909941406875 0loss/FBt1/detection_loss‚0.36023749565455065 _runtime‚10121.238064561–¾Tº:yÈ$‚(:î ×¢öÒ ²Ë) disk./.usagePercent‚76.09166634663214# disk./.usageGB‚1168.19730758667 disk.md0.out‚ 553.640625! disk.dm-0.out‚82268.107421875& disk.nvme0n1.out‚64737.12060546875 proc.memory.rssMB‚ 3024.46875& disk.nvme1n1.out‚31602.20458984375 disk.dm-0.in‚ 392.8359375" disk.nvme1c1n1.in‚ 217.30859375) proc.memory.percent‚18.45989227294922 cpu‚25.690261100319983 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 1.7749226e+07 disk.nvme1n1.in‚ 217.30859375 disk.md0.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚64737.12060546875 disk.nvme0n1.in‚ 175.52734375 network.recv‚ 2.660432e+06 disk.dm-1.out‚12921.87890625" disk.nvme0c0n1.in‚ 175.52734375( disk.nvme1c1n1.out‚31602.20458984375# memory_percent‚97.0050573348999( proc.memory.availableMB‚ 490.69140625#T0žÉ$‚(:“ ×¢öÒÞ›¦Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚264.069' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.02300000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û Þ€Ê$‚(:õ æ¢öÒÙ®’Ë$ memory_percent‚97.00372219085693( proc.memory.availableMB‚ 490.91015625 proc.memory.rssMB‚ 3024.46875 disk.nvme1n1.in‚ 217.30859375 network.sent‚ 1.7812818e+07 network.recv‚ 2.664573e+06) disk./.usagePercent‚76.09167032221633% disk./.usageGB‚1168.1973686218262" disk.nvme0c0n1.in‚ 175.52734375 disk.nvme0n1.in‚ 175.52734375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚64738.74462890625" disk.nvme1c1n1.in‚ 217.30859375" disk.dm-0.out‚82269.2548828125 proc.cpu.threads‚17& disk.nvme1n1.out‚31603.01416015625& disk.nvme0n1.out‚64738.74462890625( disk.nvme1c1n1.out‚31603.01416015625 disk.dm-0.in‚ 392.8359375) proc.memory.percent‚18.45989227294922 cpu‚25.689907569494313 disk.md0.in‚0 disk.md0.out‚ 554.1953125" disk.dm-1.out‚12921.9853515625 disk.rbd0.in‚0 disk.rbd0.out‚0»žS°žË$‚(:“ æ¢öÒ¾ãèË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚274.922' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.64066666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08ªÂ{Ì$‚(:ð õ¢öÒ«ÊÂË disk.nvme0n1.in‚ 175.52734375& disk.nvme1n1.out‚31603.70654296875) proc.memory.percent‚18.45989227294922 network.sent‚ 1.7816589e+07 disk.md0.in‚0 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 217.30859375# memory_percent‚97.0057487487793 proc.memory.rssMB‚ 3024.46875 proc.cpu.threads‚17) disk./.usagePercent‚76.09166783747621 disk.dm-0.in‚ 392.8359375 disk.dm-1.out‚12922.19140625( disk.nvme0c0n1.out‚64739.67626953125& disk.nvme0n1.out‚64739.67626953125 disk.nvme1n1.in‚ 217.30859375 network.recv‚ 2.667882e+06% disk./.usageGB‚1168.1973304748535 disk.md0.out‚ 554.5859375 disk.dm-1.in‚0( disk.nvme1c1n1.out‚31603.70654296875& proc.memory.availableMB‚ 490.578125 cpu‚25.689551773897367" disk.dm-0.out‚82269.8251953125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 175.52734375Ô%ÜžÍ$‚(:“ õ¢öÒª›Ì gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚269.705' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.90166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üï9-4Î$/ _wandbruntime‚10165  _runtime‚10165râ ÉÏ$à & lr-AdamW/pg1‚0.0002582571035191534 & lr-AdamW/pg2‚0.0002582571035191534 trainer/global_step‚11199 % _timestamp‚1.7845169916020894e+09 _runtime‚10165.177724085 _step‚919m>WÎÐ$È & lr-AdamW/pg1‚0.0002582571035191534 & lr-AdamW/pg2‚0.0002582571035191534 trainer/global_step‚11199 % _timestamp‚1.7845169916020894e+09 _runtime‚10165.177724085 _step‚919—ཤÄ4Ñ$/ _wandbruntime‚10165  _runtime‚10165Z£~L(Ò$P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1726150442224640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1726150083761152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26725234 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1716110474738384 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1703132951043024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19537548 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12977523695360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 % _timestamp‚1.7845169916032054e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46263065 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46262779 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 201360999 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90601321 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12996800059904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12996800059904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1713153283701248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110759680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1703132934003664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26725517 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚385427039183360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚372356897365504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13070160430080 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12997141484032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1726150442224640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12997141484032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1713153300740608 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1726150083761152 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚385427169356288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90601321 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110769823 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19537545 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201371144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚10165.178725767 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110759680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 _step‚920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚372356878753280 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13070271990784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12977183134086 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528‘èÒÓ$†P VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1716110117137750 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚11199 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1726150083761152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26725234 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1716110474738384 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1703132951043024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19537548 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GR ëËí"ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12977523695360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 % _timestamp‚1.7845169916032054e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46263065 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46262779 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 201360999 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90601321 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12996800059904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12996800059904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1713153283701248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110759680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1703132934003664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26725517 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚385427039183360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚372356897365504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13070160430080 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12997141484032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1726150442224640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12997141484032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1713153300740608 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1726150083761152 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚385427169356288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90601321 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110769823 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19537545 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201371144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚10165.178725767 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110759680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 _step‚920 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚372356878753280 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13070271990784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12977183134086 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1726150442224640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110769336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1713301558203392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46266756 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1713301575242752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1703280364671136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201388607 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90609112 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12978632083724 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12997910158848 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1726299468362240 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1716258656193474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1713301575242752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110779493 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201388607 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12978291522338 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1726299826832896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0kÔN2'Ö$N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1713301575242752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1703280364671136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201388607 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90609112 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12978632083724 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12997910158848 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1726299468362240 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1716258656193474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1713301575242752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110779493 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201388607 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12978291522338 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1726299826832896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19539279 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110769336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1713301558203392 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46266756 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272™Iž±4×$/ _wandbruntime‚10165  _runtime‚10165‘õIëØ$ˆ 0loss/FRt2/assignment_loss‚0.7657635531054353 0loss/FRt1/detection_loss‚0.46896250309863824 1loss/SRqqt2/detection_loss‚0.6653004887168225 % _timestamp‚1.7845169922318857e+09 _step‚922 2loss/SRqqt2/assignment_loss‚0.5026168453429352 0loss/FBt1/detection_loss‚0.38673119060004574 3loss/SRqqt1/assignment_loss‚0.47714172973122654 2loss/FBt2/assignment_loss‚0.047518787190254486 1loss/SRqqt1/detection_loss‚0.6056300067208202 epoch‚27 trainer/global_step‚11199 2loss/FBt1/assignment_loss‚0.006987978958342587 .loss/FRt2/detection_loss‚1.466932867394479 /loss/FBt2/detection_loss‚0.3241864637471735 %loss/total_loss‚7.381162248733515 _runtime‚10165.807244571 0loss/FRt1/assignment_loss‚1.6633898341273428»J"÷Ù$ 3loss/SRqqt1/assignment_loss‚0.47714172973122654 2loss/FBt2/assignment_loss‚0.047518787190254486 1loss/SRqqt1/detection_loss‚0.6056300067208202 epoch‚27 trainer/global_step‚11199 2loss/FBt1/assignment_loss‚0.006987978958342587 .loss/FRt2/detection_loss‚1.466932867394479 /loss/FBt2/detection_loss‚0.3241864637471735 %loss/total_loss‚7.381162248733515 _runtime‚10165.807244571 0loss/FRt1/assignment_loss‚1.6633898341273428 0loss/FRt2/assignment_loss‚0.7657635531054353 0loss/FRt1/detection_loss‚0.46896250309863824 1loss/SRqqt2/detection_loss‚0.6653004887168225 % _timestamp‚1.7845169922318857e+09 _step‚922 2loss/SRqqt2/assignment_loss‚0.5026168453429352 0loss/FBt1/detection_loss‚0.38673119060004574še ¡vÚ$‚(:ë „£öÒÕݳË disk.nvme0n1.in‚ 175.52734375% disk./.usageGB‚1168.1974601745605# disk.dm-1.out‚12922.40673828125 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚64748.142578125% disk.nvme1n1.out‚31607.1669921875 disk.md0.in‚0) disk./.usagePercent‚76.09167628559261 disk.dm-0.in‚ 392.8359375# disk.dm-0.out‚82278.31201171875 disk.md0.out‚ 556.15625$ memory_percent‚97.00603485107422 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922 cpu‚25.69018134825686$ disk.nvme0n1.out‚64748.142578125 network.sent‚ 1.7822547e+07 network.recv‚ 2.670978e+06 disk.dm-1.in‚0' disk.nvme1c1n1.out‚31607.1669921875% proc.memory.availableMB‚ 490.53125" disk.nvme0c0n1.in‚ 175.52734375" disk.nvme1c1n1.in‚ 217.30859375 disk.nvme1n1.in‚ 217.30859375 proc.cpu.threads‚17¶…–ŸÛ$‚(:’ „£öÒý©¡Ì gpu.0.gpu‚31.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.921' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.97366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•MsÕ4Ü$/ _wandbruntime‚10182  _runtime‚10182o“N1ÏÝ${ 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3473931550979614 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21703985333442688 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0550178624689579 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8485568761825562 -particle/accuracy_4_of_4‚0.93861323595047 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_1_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt*SRqqt*FBt/R­+¸:event_purity‚0.37890344858169556 *jet/accuracy_1_of_3‚0.9363438487052917 +jet/accuracy_3_of_3‚0.07584403455257416 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930438995361328 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2800602912902832 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7364323139190674 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6700597405433655 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8202192187309265 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4881175458431244 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2150496244430542 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6856929659843445 particle/accuracy_1_of_3‚1 trainer/global_step‚11199 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4922252893447876 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7848744988441467 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04568031057715416 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 *loss/val_total_loss‚7.5898304817364375 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7410869002342224 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8818966150283813 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9887582063674927 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.34526094794273376 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.35685425996780396 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7350568175315857 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26683589816093445 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8163644075393677 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6237737536430359 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4879027009010315 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7962962985038757 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7135105133056641 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20185279846191406 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9148934483528137 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5956193208694458 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3578135669231415 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6218595504760742 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05557481572031975 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_2‚0.8048228025436401 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.33892887830734253 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2979161739349365 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6600486040115356 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.654295802116394 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_1_of_1‚0.36652469635009766 'particle/f_score‚0.6681522727012634 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.212978795170784 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6870920658111572 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8100152015686035 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 +particle/sensitivity‚0.6578280329704285 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3774290680885315 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6890748739242554 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3184625208377838 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_2‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05111237242817879 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3473011255264282 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2863689661026001 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6336680054664612 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3648587465286255 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7042050361633301 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.16810156404972076 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3110954761505127 (particle/accuracy‚0.8596888780593872 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21104565262794495 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2705898880958557 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07955770939588547 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4866034984588623 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 particle/accuracy_1_of_1‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.988857090473175 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.372278094291687 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26165521144866943 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6520798802375793 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7705273628234863 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.8134106993675232 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 _runtime‚10182.288898886 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6861727833747864 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6232695579528809 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8100152015686035 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.15481090545654297 7validation_average_jet_accuracy‚0.40301212668418884 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.660256028175354 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6090195178985596 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17045235633850098 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8766816258430481 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.043335363268852234 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 epoch‚27 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2243947982788086 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6283682584762573 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4947916865348816 _step‚923 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6796924471855164 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8114892244338989 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.043335363268852234 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8485568761825562 /particle/accuracy_3_of_3‚0.9994120597839355 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9139502644538879 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9377617835998535 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5276535749435425 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.20250001549720764 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_1_of_5‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7304909825325012 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7699617743492126 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3471502661705017 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6512803435325623 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05147289112210274 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7048738598823547 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.35705822706222534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965382814407349 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5762499570846558 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27132317423820496 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7653425335884094 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7856786847114563 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7284636497497559 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7317708730697632 jet/accuracy_1_of_5‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6348153948783875 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.30117717385292053 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4236111044883728 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.47392967343330383 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6262364983558655 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6232695579528809 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.600318193435669 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30836158990859985 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6712496876716614 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.397830069065094 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6920576691627502 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9958748817443848 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6784283518791199 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05735323950648308 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7962962985038757 *jet/accuracy_2_of_3‚0.5956099629402161 *jet/accuracy_1_of_4‚0.9758744835853577 +jet/accuracy_3_of_4‚0.09795041382312775 $ _timestamp‚1.784517008712847e+09 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3473011255264282 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6844233274459839 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7378342747688293 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08766353130340576 Ä·5Œ=Þ$†{ =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.34526094794273376 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.35685425996780396 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7350568175315857 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26683589816093445 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8163644075393677 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6237737536430359 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4879027009010315 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7962962985038757 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7135105133056641 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20185279846191406 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9148934483528137 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5956193208694458 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3578135669231415 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6218595504760742 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05557481572031975 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_2‚0.8048228025436401 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.33892887830734253 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2979161739349365 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6600486040115356 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.654295802116394 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_1_of_1‚0.36652469635009766 'particle/f_score‚0.6681522727012634 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8"Purity/*FRt2SRqqt*FBt/event_purity‚0.212978795170784 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6870920658111572 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8100152015686035 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 +particle/sensitivity‚0.6578280329704285 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3774290680885315 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6890748739242554 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3184625208377838 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_2‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05111237242817879 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3473011255264282 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2863689661026001 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6336680054664612 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3648587465286255 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7042050361633301 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.16810156404972076 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3110954761505127 (particle/accuracy‚0.8596888780593872 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21104565262794495 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2705898880958557 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.07955770939588547 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4866034984588623 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 particle/accuracy_1_of_1‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.988857090473175 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.372278094291687 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.26165521144866943 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6520798802375793 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7705273628234863 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.8134106993675232 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 _runtime‚10182.288898886 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6861727833747864 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6232695579528809 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8100152015686035 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.15481090545654297 7validation_average_jet_accuracy‚0.40301212668418884 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.660256028175354 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6090195178985596 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.17045235633850098 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8766816258430481 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.043335363268852234 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 epoch‚27 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2243947982788086 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6283682584762573 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4947916865348816 _step‚923 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6796924471855164 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8114892244338989 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.043335363268852234 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8485568761825562 /particle/accuracy_3_of_3‚0.9994120597839355 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9139502644538879 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9377617835998535 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5276535749435425 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.20250001549720764 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_1_of_5‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7304909825325012 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7699617743492126 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3471502661705017 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6512803435325623 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05147289112210274 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7048738598823547 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.35705822706222534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965382814407349 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5762499570846558 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27132317423820496 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7653425335884094 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7856786847114563 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7284636497497559 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7317708730697632 jet/accuracy_1_of_5‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6348153948783875 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.30117717385292053 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_3‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4236111044883728 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.47392967343330383 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6262364983558655 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6232695579528809 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.600318193435669 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.30836158990859985 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6712496876716614 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.397830069065094 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6920576691627502 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9958748817443848 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6784283518791199 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05735323950648308 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7962962985038757 *jet/accuracy_2_of_3‚0.5956099629402161 *jet/accuracy_1_of_4‚0.9758744835853577 +jet/accuracy_3_of_4‚0.09795041382312775 $ _timestamp‚1.784517008712847e+09 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3473011255264282 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6844233274459839 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7378342747688293 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.08766353130340576 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3473931550979614 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21703985333442688 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0550178624689579 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8485568761825562 -particle/accuracy_4_of_4‚0.93861323595047 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_1_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.37890344858169556 *jet/accuracy_1_of_3‚0.9363438487052917 +jet/accuracy_3_of_3‚0.07584403455257416 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930438995361328 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2800602912902832 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7364323139190674 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6700597405433655 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8202192187309265 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4881175458431244 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2150496244430542 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6856929659843445 particle/accuracy_1_of_3‚1 trainer/global_step‚11199 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4922252893447876 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7848744988441467 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04568031057715416 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 *loss/val_total_loss‚7.5898304817364375 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7410869002342224 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8818966150283813 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9887582063674927›1 Ÿjxß$‚(:í “£öÒƒòÔË disk.nvme0n1.in‚ 175.52734375' disk.nvme1c1n1.out‚31853.3212890625 disk.nvme1n1.in‚ 217.30859375 disk.md0.in‚0 cpu‚25.692718613258396 network.sent‚ 1.7896785e+07) disk./.usagePercent‚76.09167479474854$ disk./.usageGB‚1168.197437286377% disk.nvme1n1.out‚31853.3212890625( proc.memory.availableMB‚ 738.48828125 proc.memory.rssMB‚ 3024.46875 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 217.30859375# disk.dm-0.out‚82278.95654296875# disk.dm-1.out‚13167.84814453125) proc.memory.percent‚18.45989227294922 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 64749.3671875" disk.nvme0n1.out‚ 64749.3671875 disk.md0.out‚ 556.76953125 disk.dm-0.in‚ 392.8359375 disk.dm-1.in‚0$ memory_percent‚95.49262523651123 proc.cpu.threads‚17 network.recv‚ 2.675553e+06" disk.nvme0c0n1.in‚ 175.52734375œgÔ×à$‚(:… “£öÒ÷åÌ gpu.0.gpu‚19.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚53.0 gpu.0.powerWatts‚119.16' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.72 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª 0V‘á$‚(:å ¢£öÒ×ÒÀË$ disk.nvme1n1.out‚31854.716796875 disk.md0.out‚ 557.421875# disk.dm-1.out‚13168.13134765625 disk.rbd0.in‚0# disk.nvme0c0n1.out‚ 64752.484375! disk.nvme0n1.out‚ 64752.484375& disk.nvme1c1n1.out‚31854.716796875 network.recv‚ 2.67992e+06) disk./.usagePercent‚76.09167901880673 disk.nvme1n1.in‚ 217.30859375 disk.md0.in‚0# disk.dm-0.out‚82281.79736328125" disk.nvme0c0n1.in‚ 175.52734375$ memory_percent‚95.49212455749512 proc.memory.rssMB‚ 3024.46875 network.sent‚ 1.7942954e+07% disk./.usageGB‚1168.1975021362305 disk.dm-0.in‚ 392.8359375 disk.dm-1.in‚0" &Ðõíßdisk.nvme1c1n1.in‚ 217.30859375' proc.memory.availableMB‚ 738.5703125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 175.52734375) proc.memory.percent‚18.45989227294922 cpu‚25.69236308627302 proc.cpu.threads‚17#ùëœâ$‚(:‘ ¢£öÒþµ‚Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.019' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.00633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€#Uœqã$‚(:æ ±£öÒ×ùÂË% disk./.usageGB‚1168.1980285644531 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 217.30859375% disk.nvme1c1n1.out‚32100.75390625 network.sent‚ 1.7947782e+07 network.recv‚ 2.68357e+06& disk.nvme0c0n1.out‚64753.755859375 disk.nvme1n1.in‚ 217.30859375# disk.dm-0.out‚82282.65283203125' proc.memory.availableMB‚ 738.5390625 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922# disk.dm-1.out‚13413.61962890625 disk.nvme0n1.in‚ 175.52734375$ disk.nvme0n1.out‚64753.755859375 disk.md0.in‚0 disk.dm-0.in‚ 392.8359375# memory_percent‚95.4923152923584 proc.cpu.threads‚17) disk./.usagePercent‚76.09171330822036 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 175.52734375# disk.nvme1n1.out‚32100.75390625 disk.md0.out‚557.875 cpu‚25.691983220392352ï6;žä$‚(:“ ±£öÒéÙƒÌ gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚269.719' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.90633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì"4å$/ _wandbruntime‚10227  _runtime‚10227œA¿SÇæ$Á % lr-AdamW/pg1‚0.000255827861455034 % lr-AdamW/pg2‚0.000255827861455034 trainer/global_step‚11249 % _timestamp‚1.7845170537369645e+09 _runtime‚10227.312985177 _step‚924ÒéäIÌç$Æ % lr-AdamW/pg1‚0.000255827861455034 % lr-AdamW/pg2‚0.000255827861455034 trainer/global_step‚11249 % _timestamp‚1.7845170537369645e+09 _runtime‚10227.312985177 _step‚924œ’~`‘4è$/ _wandbruntime‚10227  _runtime‚10227¯BpR(é$þO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26873450 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13053310757670 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13072687000064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111335974 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1712425740424848 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111325831 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19628361 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91113779 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1735582039684096 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1722509369723392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1725479051182518 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13073028424192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111325831 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚387185178527232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚374038470919680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚10227.313391418 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1722509352684032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19628364 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202449753 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13146707607552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13072687000064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111335974 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1712425740424848 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111325831 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19628361 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91113779 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1735582039684096 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1722509369723392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1725479051182518 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13073028424192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111325831 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚387185178527232 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚374038470919680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚10227.313391418 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1722509352684032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19628364 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202449753 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13146707607552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1flR4ë$/ _wandbruntime‚10227  _runtime‚10227:‹&Næ%ì$ŠN SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111335487 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1712573154052960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13054419146034 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111335487 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26875773 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13074138530304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19630098 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1722657644225536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13073797099008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚11249 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1722657644225536 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1722657627186176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26875485 % _timestamp‚1.7845170543654673e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46505580 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374071651189760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1722657627186176 _step‚926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1735731782755840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91121570 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202467216 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91121572 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46505871 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202457057 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1735731782755840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19630095 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13054078584648 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91121572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202457057 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _runtime‚10227.941239599 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚387219477646336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91121570 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202467216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0âWÒë'í$N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19630098 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1722657644225536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13073797099008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚11249 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1722657644225536 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1722657627186176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26875485 % _timestamp‚1.7845170543654673e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46505580 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚374071651189760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1722657627186176 _step‚926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1735731782755840 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91121570 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202467216 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91121572 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46505871 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202457057 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1735731782755840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19630095 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13054078584648 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91121572 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202457057 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _runtime‚10227.941239599 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚387219477646336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91121570 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202467216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111335487 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1712573154052960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13054419146034 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111335487 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26875773 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13074138530304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ž;]u@4î$/ _wandbruntime‚10227  _runtime‚10227#ü ï$‡ /loss/FBt1/detection_loss‚0.2504260754935315 /loss/FBt2/detection_loss‚0.5868209240814815 _runtime‚10227.945013086 /loss/FRt1/assignment_loss‚1.608878806053874 2loss/SRqqt2/assignment_loss‚0.4966444980043964 1loss/SRqqt2/detection_loss‚0.6155156836415945 0loss/FRt2/assignment_loss‚0.6883940401096498 2loss/FBt2/assignment_loss‚0.006572033911852989 0loss/FRt1/detection_loss‚0.47023620969842245 %loss/total_loss‚7.334788043271367 epoch‚28 trainer/global_step‚11249 % _timestamp‚1.7845170543665116e+09 _step‚927 2loss/SRqqt1/assignment_loss‚0.4421794578582138 /loss/FRt2/detection_loss‚1.3895718292408292 1loss/SRqqt1/detection_loss‚0.7784489688334557 3loss/FBt1/assignment_loss‚0.0010995163440644777F¼€Çð$Œ % _timestamp‚1.7845170543665116e+09 _step‚927 2loss/SRqqt1/assignment_loss‚0.4421794578582138 /loss/FRt2/detection_loss‚1.3895718292408292 1loss/SRqqt1/detection_loss‚0.7784489688334557 3loss/FBt1/assignment_loss‚0.0010995163440644777 /loss/FBt1/detection_loss‚0.2504260754935315 /loss/FBt2/detection_loss‚0.5868209240814815 _runtime‚10227.945013086 /loss/FRt1/assignment_loss‚1.608878806053874 2loss/SRqqt2/assignment_loss‚0.4966444980043964 1loss/SRqqt2/detection_loss‚0.6155156836415945 0loss/FRt2/assignment_loss‚0.6883940401096498 2loss/FBt2/assignment_loss‚0.006572033911852989 0loss/FRt1/detection_loss‚0.47023620969842245 %loss/total_loss‚7.334788043271367 epoch‚28 trainer/global_step‚11249ŸfÎsñ$‚(:è À£öÒÙÖË disk.md0.out‚ 558.00390625 disk.dm-0.out‚ 82283.96875 cpu‚25.691579376167347 proc.cpu.threads‚17 disk.nvme0n1.in‚ 175.52734375' disk.nvme1c1n1.out‚32101.3876953125 network.recv‚ 2.686699e+06 disk.rbd0.out‚0 disk.dm-0.in‚ 392.8359375# memory_percent‚95.4925537109375 proc.memory.rssMB‚ 3024.46875 network.sent‚ 1.7963595e+07% disk./.usageGB‚1168.1984519958496& disk.nvme0n1.out‚64754.92724609375 disk.md0.in‚0! proc.memory.availableMB‚738.5) proc.memory.percent‚18.45989227294922" disk.nvme1c1n1.in‚ 217.30859375) disk./.usagePercent‚76.09174088883567 disk.dm-1.in‚0# disk.dm-1.out‚13413.81982421875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 175.52734375( disk.nvme0c0n1.out‚64754.92724609375 disk.nvme1n1.in‚ 217.30859375% disk.nvme1n1.out‚32101.3876953125ˆFü|žò$‚(:“ À£öÒ˜Ð–Ì gpu.0.gpu‚64.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.559' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.18633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-eshó$‚(:Ý Ï£öÒıË proc.memory.rssMB‚ 3024.46875% disk./.usageGB‚1168.1992721557617 disk.dm-0.in‚ 392.8359375 disk.dm-0.out‚ 82284.9296875 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚64756.30517578125( proc.memory.availableMB‚ 738.50390625 proc.cpu.threads‚17 network.sent‚ 1.8013126e+07( disk./.usagePercent‚76.0917943107482" disk.nvme1c1n1.in‚ 217.30859375 disk.nvme1n1.in‚ 217.30859375 disk.nvme1n1.out‚ 32102.375$ memory_percent‚95.49252986907959# disk.dm-1.out‚13413.90576171875 disk.rbd0.in‚0 disk.nvme1c1n1.out‚ 32102.375 disk.md0.in‚0 disk.md0.out‚558.625) proc.memory.percent‚18.45989227294922 cpu‚25.69120337720673 network.recv‚ 2.690933e+06" disk.nvme0c0n1.in‚ 175.52734375 disk.nvme0n1.in‚ 175.52734375& disk.nvme0n1.out‚64756.30517578125E½Vwžô$‚(:“ Ï£öÒÒÀˆÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚272.891' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.96366666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"™Šƒõ$‚(:ô Þ£öÒŠÞÀË network.recv‚ 2.694067e+06! disk.dm-0.out‚82286.728515625& disk.nvme0n1.out‚64758.27001953125 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922 cpu‚25.690826350228082) disk./.usagePercent‚76.09179480769622 disk.nvme1n1.in‚ 217.30859375 disk.md0.out‚ 559.0390625 disk.dm-0.in‚ 392.8359375" disk.dm-1.out‚13414.0400390625( disk.nvme0c0n1.out‚64758.27001953125" disk.nvme1c1n1.in‚ 217.30859375( disk.nvme1c1n1.out‚32103.22998046875 network.sent‚ 1.8019909e+07% disk./.usageGB‚1168.1992797851562 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 175.52734375 disk.nvme0n1.in‚ 175.52734375$ memory_percent‚95.49252986907959& disk.nvme1n1.out‚32103.22998046875 disk.rbd0.in‚0( proc.memory.availableMB‚ 738.50390625 proc.cpu.threads‚17‰xåxžö$‚(:“ Þ£öÒ¢ÔƒÌ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.093' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.36433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø]Q4÷$/ _wandbruntime‚10271  _runtime‚10271®†a{Ëø$Å _step‚928 ' lr-AdamW/pg1‚0.00025340192179907673 ' lr-AdamW/pg2‚0.00025340192179907673 trainer/global_step‚11299 % _timestamp‚1.7845170982834609e+09 _runtime‚10271.859077652{à$Ðù$Ê _step‚928 ' lr-AdamW/pg1‚0.00025340192179907673 ' lr-AdamW/pg2‚0.00025340192179907673 trainer/global_step‚11299 % _timestamp‚1.7845170982834609e+09 _runtime‚10271.859077652 x=8â4ú$/ _wandbruntime‚10271  _runtime‚10271¯˜ë(û$ÿO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91503429 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91503427 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13128597526016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26988262 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _runtime‚10271.860205363 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1743051333893120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13128256101888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92686336 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1732905710168892 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13108453409804 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91503429 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚388900163978752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13202568256512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13202660942848 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203313910 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91503427 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13108793971078 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111810483 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111810483 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚388900052680192 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184Ô÷f»ü$„P TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91503429 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91503427 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13128597526016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26988262 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _runtime‚10271.860205363 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1743051333893120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsü«OMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13128256101888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92686336 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1732905710168892 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13108453409804 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91503429 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚388900163978752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13202568256512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13202660942848 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203313910 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91503427 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13108793971078 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111810483 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111810483 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚388900052680192 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496¡>4ý$/ _wandbruntime‚10272  _runtime‚10272Öé'þ$‹N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1730071352293376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 _runtime‚10272.485516243 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13129366200832 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111830296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚388934465685504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚375730683305984 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1719944687426560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111830296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203331359 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1730071369332736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13129707632128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1730071352293376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13129366200832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26990597 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13203782379520 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19716795 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46707392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1719944704465920 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203341518 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388934346005504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13203681311744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚930 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111820139 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1733054249224616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26990852 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1730071369332736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚375730664693760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19716798 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203341518 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1743201076964864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111820139 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 trainer/global_step‚11299 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203341518 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1743201076964864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111820139 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 trainer/global_step‚11299 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚375730683305984 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1719944687426560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111830296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203331359 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1730071369332736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13129707632128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1730071352293376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13129366200832 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26990597 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13203782379520 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19716795 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46707392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1719944704465920 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203341518 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388934346005504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13203681311744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚930 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111820139 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1733054249224616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26990852 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1730071369332736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚375730664693760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19716798 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1727168790132688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204198357 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390615041050112 \ODeviceStatsMonitor.on_train_batcAÈ~–³h_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1737336802898432 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1750520628102144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27103278 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1737336819937792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13164277184486 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112305278 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 136464384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390614904585728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13258524510208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452û9# (Ž%„P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46905345 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46905039 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112305278 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1750520628102144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13183825203712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1727168790132688 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204198357 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390615041050112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1737336802898432 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1750520628102144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27103278 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1737336819937792 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13164277184486 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112305278 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 136464384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390614904585728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13258524510208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1737336819937792 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13183825203712 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1750520986565632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91893079 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452¥#ͦú4%/ _wandbruntime‚10317  _runtime‚10317k„F'%‹N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1737485077400576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚377389696809984 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13165385572850 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13259633683456 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19803495 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _runtime‚10317.046283114 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204205661 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112314948 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1727316237839520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚934 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204215820 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1750670371173888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390649204521984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13184935302656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112304791 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1740481265811624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13165045011464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19803498 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91900870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1727316220800160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112314948 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204205661 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390649330493440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1740481623412370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13185276733952 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1750670012703232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46909076 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚377389678197760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91900870 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27105581 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1737485094439936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46909374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1737485077400576 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204215820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27105876 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚11349 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91900872¢ÊSPÛ‘%N QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _runtime‚10317.046283114 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204205661 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112314948 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1727316237839520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚934 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204215820 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1750670371173888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390649204521984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13184935302656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112304791 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1740481265811624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13165045011464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19803498 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91900870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_bÓíÛó;atch_end/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1727316220800160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112314948 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204205661 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390649330493440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1740481623412370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13185276733952 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1750670012703232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46909076 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚377389678197760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91900870 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27105581 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1737485094439936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46909374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1737485077400576 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204215820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27105876 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚11349 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91900872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1737485077400576 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚377389696809984 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13165385572850 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13259633683456 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19803495 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180¦X~7\4’%/ _wandbruntime‚10317  _runtime‚10317Tj*“%ˆ 1loss/SRqqt1/detection_loss‚0.6535024209386002 % _timestamp‚1.7845171434716082e+09 0loss/FRt1/assignment_loss‚1.5896657022005491 2loss/FBt1/assignment_loss‚0.001142278155192246 2loss/FBt2/assignment_loss‚0.001339770868720791 0loss/FRt1/detection_loss‚0.47250674445143137 1loss/SRqqt2/detection_loss‚0.6287247690621244 0loss/FRt2/assignment_loss‚0.7191497735130838 2loss/SRqqt2/assignment_loss‚0.4736106326108567 0loss/FBt1/detection_loss‚0.34986910498035806 epoch‚28 _runtime‚10317.046886295 /loss/FBt2/detection_loss‚0.3804262369250258 %loss/total_loss‚7.215019579713201 trainer/global_step‚11349 _step‚935 2loss/SRqqt1/assignment_loss‚0.5066581360449565 /loss/FRt2/detection_loss‚1.4384240099623018ÛO ”% 2loss/SRqqt1/assignment_loss‚0.5066581360449565 /loss/FRt2/detection_loss‚1.4384240099623018 1loss/SRqqt1/detection_loss‚0.6535024209386002 % _timestamp‚1.7845171434716082e+09 0loss/FRt1/assignment_loss‚1.5896657022005491 2loss/FBt1/assignment_loss‚0.001142278155192246 2loss/FBt2/assignment_loss‚0.001339770868720791 0loss/FRt1/detection_loss‚0.47250674445143137 1loss/SRqqt2/detection_loss‚0.6287247690621244 0loss/FRt2/assignment_loss‚0.7191497735130838 2loss/SRqqt2/assignment_loss‚0.4736106326108567 0loss/FBt1/detection_loss‚0.34986910498035806 epoch‚28 _runtime‚10317.046886295 /loss/FBt2/detection_loss‚0.3804262369250258 %loss/total_loss‚7.215019579713201 trainer/global_step‚11349 _step‚935§ùÙyv•%‚(:ë š¤öÒþ¾âË disk.md0.out‚ 562.69921875$ memory_percent‚95.49312591552734 proc.memory.rssMB‚ 3024.46875( disk./.usagePercent‚76.0917244895509# disk./.usageGB‚1168.19820022583" disk.nvme1c1n1.in‚ 227.71484375! disk.dm-0.out‚82295.986328125 disk.dm-1.out‚ 13414.609375( disk.nvme0c0n1.out‚64769.43798828125 disk.nvme0n1.in‚ 292.37109375& disk.nvme0n1.out‚64769.43798828125& disk.nvme1n1.out‚32109.49853515625 disk.dm-0.in‚ 520.0859375 disk.dm-1.in‚0 cpu‚25.689354455858304 proc.cpu.threads‚17( disk.nvme1c1n1.out‚32109.49853515625 disk.nvme1n1.in‚ 227.71484375 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0% proc.memory.availableMB‚ 738.40625) proc.memory.percent‚18.45989227294922 network.sent‚ 1.8107999e+07 network.recv‚ 2.708518e+06" disk.nvme0c0n1.in‚ 292.37109375…Ôöí“–%‚(:ˆ š¤öÒ„’¢Ì gpu.0.gpu‚79.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.432' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.144 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CJn—%‚(:ã ©¤öҠѯË& disk.nvme1n1.out‚32110.18505859375 disk.md0.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚76.09172821666107 disk.nvme0n1.in‚ 292.828125 disk.rbd0.out‚0$ memory_percent‚95.49312591552734% proc.memory.availableMB‚ 738.40625) proc.memory.percent‚18.45989227294922 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 292.828125 disk.nvme1n1.in‚ 228.828125 disk.md0.out‚ 563.16796875 disk.dm-0.in‚ 521.65625! disk.dm-0.out‚82296.697265625 disk.dm-1.in‚0 disk.dm-1.out‚ 13414.7109375 proc.memory.rssMB‚ 3024.46875 network.sent‚ 1.8159746e+07$ disk./.usageGB‚1168.198257446289( disk.nvme0c0n1.out‚64770.55419921875 disk.nvme1c1n1.in‚ 228.828125 cpu‚25.68898552809166 network.recv‚ 2.713411e+06& disk.nvme0n1.out‚64770.55419921875( disk.nvme1c1n1.out‚32110.18505859375ñA™o˜%‚(:’ ©¤öÒ¨”ýË gpu.0.gpu‚17.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.638' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.21266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aP À|™%‚(:ñ ¸¤öÒªîÎË% proc.memory.availableMB‚ 738.40625 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922" disk.nvme0c0n1.in‚ 293.73828125 cpu‚25.68859020227469 disk.dm-0.in‚ 522.56640625# disk.dm-1.out‚13414.88525390625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚64772.18505859375 disk.nvme1c1n1.in‚ 228.828125( disk.nvme1c1n1.out‚32110.85986328125 disk.md0.in‚0 proc.cpu.threads‚17 network.sent‚ 1.8164574e+07% disk./.usageGB‚1168.1983337402344# disk.dm-0.out‚82298.29638671875 disk.dm-1.in‚0& disk.nvme0n1.out‚64772.18505859375& disk.nvme1n1.out‚32110.85986328125 disk.md0.out‚ 563.41015625$ memory_percent‚95.49312591552734 network.recv‚ 2.716512e+06) disk./.usagePercent‚76.09173318614131 disk.rbd0.out‚0 disk.nvme0n1.in‚ 293.73828125 disk.nvme1n1.in‚ 228.828125òáèžš%‚(:“ ¸¤öÒÎ¥ŒÌ gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.638' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.54599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jµP¬4›%/ _wandbruntime‚10360  _runtime‚10360jâ÷ëÈœ% & lr-AdamW/pg1‚0.0002485605727438168 & lr-AdamW/pg2‚0.0002485605727438168 trainer/global_step‚11399 % _timestamp‚1.7845171870856512e+09 _runtime‚10360.66128392 _step‚936uº¬Í%Ç % _timestamp‚1.7845171870856512e+09 _runtime‚10360.66128392 _step‚936 & lr-AdamW/pg1‚0.0002485605727438168 & lr-AdamW/pg2‚0.0002485605727438168 trainer/global_step‚11399¨7„wø4ž%/ _wandbruntime‚10360  _runtime‚10360¯DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13314336507392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92282727 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1757989922311168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1757990280774656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27217711 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13314231238144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112779787 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1744750528005632 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚392329866551296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1757990280774656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379015511431680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13219760397894 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112789930 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112779787 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19888464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19888461 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.784517187086765e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13239394305536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92282729 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚10360.662335842 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1757989922311168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92282727 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13239735729664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13219419836620 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205072659 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚379015530043904A²(J ( %…P ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1757990280774656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27217711 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13314231238144 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112779787 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1744750528005632 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚392329866551296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1757990280774656 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379015511431680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13219760397894 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112789930 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112779787 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19888464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19888461 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.784517187086765e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13239394305536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92282729 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚10360.662335842 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1757989922311168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92282727 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13239735729664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13219419836620 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205072659 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚379015530043904 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13239394305536 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47106172 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1734540340545648 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1744750545044992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13314336507392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92282727 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1757989922311168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360©ÕfßÒ4¡%/ _wandbruntime‚10361  _runtime‚10361¶<–p‚¢%ŠN FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47110175 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1744898819547136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚392364164745728 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13315349169664 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1734687771213120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205090122 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚379048691701760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19890195 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 trainer/global_step‚11399 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205079963 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1744898802507776 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1758139665382912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92290520 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13240845835776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13240504404480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27220266 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784517187711468e+09 _runtime‚10361.287265706 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47110464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1747908282398632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112799600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392364040871424 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13315454431744 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1758139665382912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13220528224872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205090122 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205079963 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92290522 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ÏüÈø'£%N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1734687754173760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13240845835776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13240504404480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27220266 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784517187711468e+09 _runtime‚10361.287265706 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47110464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1747908282398632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112799600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392364040871424 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13315454431744 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1758139665382912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13220528224872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205090122 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205079963 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92290522 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47110175 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1744898819547136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚392364164745728 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13315349169664 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1734687771213120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205090122 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚379048691701760 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19890195 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 trainer/global_step‚11399 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205079963 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1744898802507776 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1758139665382912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92290520 Bͯ%Ç & lr-AdamW/pg2‚0.0002461454742953172 trainer/global_step‚11449 $ _timestamp‚1.784517231581021e+09 _runtime‚10405.156994264 _step‚940 & lr-AdamW/pg1‚0.0002461454742953172¬2'ÜO4°%/ _wandbruntime‚10405  _runtime‚10405¹Üli“±%ÿO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13370100379648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47307235 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19975161 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1765459216520192 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1752164270152192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1741911856879888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92672377 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _runtime‚10405.157289475 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13274903050028 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1752164253112832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113264439 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92672379 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1765459574983680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380674543547904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92672377 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1755187117530550 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47307235 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19975161 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1765459216520192 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1752164270152192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1741911856879888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92672377 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _runtime‚10405.157289475 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13274903050028 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1752164253112832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113264439 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92672379 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1765459574983680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚380674543547904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0­Æ5îÊ4³%/ _wandbruntime‚10405  _runtime‚10405±Z0'´%‹N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205954265 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1742059304586720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205964424 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13371321908224 _runtime‚10405.782640162 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13296073506304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113284252 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27334644 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13276351999666 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13276011438280 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27334362 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92680170 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7845172322068913e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47311257 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1752312544654336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205954265 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92680172 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92680172 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1765608959591936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380707723817984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380707705205760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1755335656586386 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19976895 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1755335298985640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1765608959591936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113284252 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1752312544654336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚942 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205964424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1765608601121280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394079045726208ˆˆz®%µ%N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205954265 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1742059304586720 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205964424 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13371321908224 _runtime‚10405.782640162 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13296073506304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113284252 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27334644 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13276351999666 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13276011438280 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27334362 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92680170 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7845172322068913e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47311257 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1752312544654336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205954265 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92680172 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92680172 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1765608959591936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380707723817984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚380707705205760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1755335656586386 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19976895 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1755335298985640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1765608959591936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113284252 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1752312544654336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚942 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205964424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 T?DeviceStatsMonitor.on_train_batch_end/allocated_j\—úbytes.all.freed‚1765608601121280 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394079045726208 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360®›Éž4¶%/ _wandbruntime‚10405  _runtime‚10405«-E ·%„ _runtime‚10405.78323366 0loss/FRt1/assignment_loss‚1.5661778286127497 2loss/SRqqt1/assignment_loss‚0.4498026470539942 3loss/SRqqt2/assignment_loss‚0.48672049316842025 2loss/FBt1/assignment_loss‚0.013185815447943127 /loss/FRt1/detection_loss‚0.4470126129599589 .loss/FRt2/detection_loss‚1.486037531833416 1loss/SRqqt1/detection_loss‚0.6794065140724264 0loss/FBt1/detection_loss‚0.41798804091582686 $ _timestamp‚1.784517232207938e+09 0loss/FRt2/assignment_loss‚0.7918516670075638 %loss/total_loss‚7.715410552211959 _step‚943 1loss/FBt2/assignment_loss‚0.17689769046738127 1loss/SRqqt2/detection_loss‚0.6564541148310084 /loss/FBt2/detection_loss‚0.5438755958412702 epoch‚28 trainer/global_step‚11449»¾›¸%‰ 0loss/FRt2/assignment_loss‚0.7918516670075638 %loss/total_loss‚7.715410552211959 _step‚943 1loss/FBt2/assignment_loss‚0.17689769046738127 1loss/SRqqt2/detection_loss‚0.6564541148310084 /loss/FBt2/detection_loss‚0.5438755958412702 epoch‚28 trainer/global_step‚11449 _runtime‚10405.78323366 0loss/FRt1/assignment_loss‚1.5661778286127497 2loss/SRqqt1/assignment_loss‚0.4498026470539942 3loss/SRqqt2/assignment_loss‚0.48672049316842025 2loss/FBt1/assignment_loss‚0.013185815447943127 /loss/FRt1/detection_loss‚0.4470126129599589 .loss/FRt2/detection_loss‚1.486037531833416 1loss/SRqqt1/detection_loss‚0.6794065140724264 0loss/FBt1/detection_loss‚0.41798804091582686 $ _timestamp‚1.784517232207938e+09¯5ôƒy¹%‚(:î ô¤öÒóÝëË proc.cpu.threads‚17 network.sent‚ 1.8251779e+07 network.recv‚ 2.730091e+06& disk.nvme0c0n1.out‚65811.623046875" disk.nvme1c1n1.in‚ 229.51953125# disk.dm-0.out‚83336.93798828125 disk.dm-1.in‚0 cpu‚25.687168060115397% disk./.usageGB‚1169.2055320739746 disk.dm-0.in‚ 525.0234375 disk.rbd0.in‚0( proc.memory.availableMB‚ 738.30859375 proc.memory.rssMB‚ 3024.46875 disk.md0.out‚ 565.2109375" disk.nvme0c0n1.in‚ 295.50390625 disk.nvme0n1.in‚ 295.50390625$ disk.nvme1n1.out‚32114.248046875 disk.md0.in‚0 disk.rbd0.out‚0# memory_percent‚95.4937219619751) disk./.usagePercent‚76.15733802794168$ disk.nvme0n1.out‚65811.623046875& disk.nvme1c1n1.out‚32114.248046875 disk.nvme1n1.in‚ 229.51953125# disk.dm-1.out‚13415.21923828125) proc.memory.percent‚18.45989227294922±J•žº%‚(:“ ô¤öÒû•³Ì gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.204' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.73466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–*w»%‚(:ì ƒ¥öÒˆá¶Ë disk.md0.in‚0 disk.rbd0.in‚0) proc.memory.percent‚18.45989227294922 cpu‚25.686781587422455 proc.cpu.threads‚17 network.recv‚ 2.734564e+06% disk./.usageGB‚1169.2097816467285 disk.dm-0.in‚525.75# disk.dm-1.out‚13415.33642578125" disk.nvme0c0n1.in‚ 295.87890625# memory_percent‚95.4937219619751( proc.memory.availableMB‚ 738.30859375) disk./.usagePercent‚76.15761482799078" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 565.515625 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚65812.7685546875 proc.memory.rssMB‚ 3024.46875 network.sent‚ 1.8303259e+07% disk.nvme0n1.out‚65812.7685546875' disk.nvme1c1n1.out‚32114.7958984375# disk.dm-0.out‚83337.83251953125 disk.nvme0n1.in‚ 295.87890625% disk.nvme1n1.out‚32114.7958984375Ôþ Ó¼%‚(:ˆ ƒ¥öÒŒÛƒÍ gpu.0.gpu‚92.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.916' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í+˜rs½%‚(:è ’¥öÒáãÑË" disk.nvme0c0n1.in‚ 295.87890625& disk.nvme0c0n1.out‚65817.044921875) disk./.usagePercent‚76.15863058975087% disk.nvme1n1.out‚32115.4443359375" disk.dm-1.out‚13415.4345703125' disk.nvme1c1n1.out‚32115.4443359375 proc.memory.rssMB‚ 3024.46875) proc.memory.percent‚18.45989227294922 cpu‚25.686919006113442 network.sent‚ 1.8308078e+07 disk.nvme0n1.in‚ 295.87890625" disk.nvme1c1n1.in‚ 229.87109375( proc.memory.availableMB‚ 738.30859375 proc.cpu.threads‚17% disk./.usageGB‚1169.2253761291504 disk.md0.in‚0 disk.dm-0.in‚525.75 disk.dm-1.in‚0$ disk.nvme0n1.out‚65817.044921875# memory_percent‚95.4937219619751 network.recv‚ 2.737934e+06 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 565.91796875 disk.dm-0.out‚83341.76953125 disk.rbd0.in‚0 disk.rbd0.out‚0SÎ0“¾%‚(:ˆ ’¥öÒå…Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.581' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.527 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?Ð]4¿%/ _wandbruntime‚10448  _runtime‚10448”|kËÀ%Å _runtime‚10448.818573112 _step‚944 ' lr-AdamW/pg1‚0.00024373430013661392 ' lr-AdamW/pg2‚0.00024373430013661392 trainer/global_step‚11499 % _timestamp‚1.7845172752429588e+096LÂÐÁ%Ê ' lr-AdamW/pg1‚0.00024373430013661392 ' lr-AdamW/pg2‚0.00024373430013661392 trainer/global_step‚11499 % _timestamp‚1.7845172752429588e+09 _runtime‚10448.818573112 _step‚944°^g±‡4Â%/ _wandbruntime‚10448  _runtime‚104484Äu|(Ã%P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1772928510729216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113759234 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1749283390253488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1759577978220032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47508263 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20061864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13425938328064 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206821263 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚395759476767744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1759577978220032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20061861 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13330726824710 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93062029 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1759577995259392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206811118 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27446399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1749283407292848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93062027 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚10448.819768553 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184äU½ (Ä%†P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1772928510729216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113759234 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1749283390253488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1759577978220032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47508263 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20061864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13425938328064 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206821263 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚395759476767744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1759577978220032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20061861 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13330726824710 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93062029 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1759577995259392 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206811118 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27446399 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1749283407292848 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93062027 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚10448.819768553 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47512284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20063598 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13351642608128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93069822 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1749430837960320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113768904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚11499 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1773078253800960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47511995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1773078253800960 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1759726269761536 S9F|5GEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚395793765767680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚382366718709760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚395793902224896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13331494651688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13331835213074 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13351984039424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1773077895330304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 206828567 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 206838726 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206828567 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13351984039424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206838726 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113758747˜ŒÈm'Ç%N J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93069820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1759726252722176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206838726 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113758747 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93069822 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47512284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20063598 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13351642608128 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93069822 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1749430837960320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113768904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚11499 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1773078253800960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47511995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1773078253800960 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1759726269761536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚395793765767680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚382366718709760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚395793902224896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13331494651688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13331835213074 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13351984039424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1773077895330304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 206828567 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 206838726 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206828567 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13351984039424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 BÉ%ˆ % _timestamp‚1.7845172758586335e+09 _runtime‚10449.433907948 3loss/SRqqt1/assignment_loss‚0.42687582788216244 3loss/SRqqt2/assignment_loss‚0.45053757287810825 3loss/FBt1/assignment_loss‚0.0005062133607003274 1loss/FBt2/assignment_loss‚0.00841622013736138 1loss/SRqqt1/detection_loss‚0.6213952912949026 /loss/FBt2/detection_loss‚0.8116585349874391 &loss/total_loss‚7.8813161929256115 epoch‚28 1loss/SRqqt2/detection_loss‚0.6947840177748394 trainer/global_step‚11499 _step‚947 /loss/FRt1/assignment_loss‚1.579711956786923 /loss/FRt2/detection_loss‚1.5407227599462712 0loss/FRt2/assignment_loss‚0.8397663791512334 0loss/FRt1/detection_loss‚0.46294021774038896 .loss/FBt1/detection_loss‚0.444001200985281ë1Ê% _step‚947 /loss/FRt1/assignment_loss‚1.579711956786923 /loss/FRt2/detection_loss‚1.5407227599462712 0loss/FRt2/assignment_loss‚0.8397663791512334 0loss/FRt1/detection_loss‚0.46294021774038896 .loss/FBt1/detection_loss‚0.444001200985281 % _timestamp‚1.7845172758586335e+09 _runtime‚10449.433907948 3loss/SRqqt1/assignment_loss‚0.42687582788216244 3loss/SRqqt2/assignment_loss‚0.45053757287810825 3loss/FBt1/assignment_loss‚0.0005062133607003274 1loss/FBt2/assignment_loss‚0.00841622013736138 1loss/SRqqt1/detection_loss‚0.6213952912949026 /loss/FBt2/detection_loss‚0.8116585349874391 &loss/total_loss‚7.8813161929256115 epoch‚28 1loss/SRqqt2/detection_loss‚0.6947840177748394 trainer/global_step‚11499³MëÒlË%‚(:á ¡¥öÒ¿‚ÌË disk.md0.in‚0 disk.dm-0.out‚83414.31640625 disk.dm-1.in‚0" disk.dm-1.out‚13415.5244140625 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 295.87890625& disk.nvme0n1.out‚65889.89111328125 disk.nvme1n1.in‚ 229.87109375) disk./.usagePercent‚76.16713014027134% disk./.usageGB‚1169.3558654785156 disk.md0.out‚566.25( disk.nvme1c1n1.out‚32115.95849609375& disk.nvme1n1.out‚32115.95849609375" proc.memory.availableMB‚493.75 proc.memory.rssMB‚ 3024.46875 proc.cpu.threads‚17 disk.rbd0.in‚0( disk.nvme0c0n1.out‚65889.89111328125" disk.nvme1c1n1.in‚ 229.87109375) proc.memory.percent‚18.45989227294922 cpu‚25.68681861273901 network.sent‚ 1.8323939e+07 network.recv‚ 2.74136e+06 disk.dm-0.in‚525.75 disk.nvme0n1.in‚ 295.87890625$ memory_percent‚96.98638916015625‹eŒ¥œÌ%‚(:‘ ¡¥öÒ¾ÀÌ gpu.0.gpu‚27.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚289.02' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.33999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±$ô+xÍ%‚(:í °¥öÒòò©Ë) disk./.usagePercent‚76.16710554134417 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚66002.400390625$ disk.nvme0n1.out‚66002.400390625" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 566.43359375( proc.memory.availableMB‚ 493.73828125 network.sent‚ 1.8375482e+07 network.recv‚ 2.745311e+06% disk./.usageGB‚1169.3554878234863 disk.dm-1.in‚0# disk.dm-1.out‚13415.64697265625' disk.nvme1c1n1.out‚32116.8466796875 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚32116.8466796875 disk.md0.in‚0 proc.memory.rssMB‚ 3024.46875 cpu‚25.686457377089084" disk.nvme0c0n1.in‚ 295.87890625 disk.nvme0n1.in‚ 295.87890625 disk.dm-0.in‚525.75# disk.dm-0.out‚83527.17724609375$ memory_percent‚96.98646068572998) proc.memory.percent‚18.45989227294922 proc.cpu.threads‚17jŸq5Î%‚(:’ °¥öÒ ŠëË gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚243.92' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚81.30666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ô±(rÏ%‚(:ç ¿¥öÒÏË" disk.nvme0n1.out‚ 66046.5703125$ memory_percent‚96.98648452758789 cpu‚25.686141945796635 proc.cpu.threads‚17 network.recv‚ 2.748829e+06' disk.nvme1c1n1.out‚32117.5869140625 disk.md0.in‚0 disk.md0.out‚ 566.92578125 disk.dm-0.in‚525.75 disk.dm-1.in‚0# disk.dm-1.out‚13415.69384765625" disk.nvme0c0n1.in‚ 295.87890625% disk./.usageGB‚1169.3560218811035" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375# disk.dm-0.out‚83570.99755859375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 295.87890625 proc.memory.rssMB‚ 3024.46875) disk./.usagePercent‚76.16714032770582$ disk.nvme0c0n1.out‚ 66046.5703125& proc.memory.availableMB‚ 493.734375) proc.memory.percent‚18.45989227294922 network.sent‚ 1.8378404e+07% disk.nvme1n1.out‚32117.5869140625¡‹'žÐ%‚(:“ ¿¥öÒóÙ¦Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚185.318' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚61.77266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜Z:{4Ñ%/ _wandbruntime‚10490  _runtime‚10490rõê|ËÒ%Å trainer/global_step‚11549 % _timestamp‚1.7845173173290393e+09 _runtime‚10490.906014219 _step‚948 ' lr-AdamW/pg1‚0.00024132720513291768 ' lr-AdamW/pg2‚0.00024132720513291768£‘ÐÓ%Ê ' lr-AdamW/pg1‚0.00024132720513291768 ' lr-AdamW/pg2‚0.00024132720513291768 trainer/global_step‚11549 % _timestamp‚1.7845173173290393e+09 _runtime‚10490.906014219 _step‚948´Åk/4Ô%/ _wandbruntime‚10490  _runtime‚10490åhL(Õ%ÿO VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1780397804938240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13406101611008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚397474483527168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1766991703327232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13406443035136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93451679 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1770041150704566 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13385869476844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27561082 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207695565 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚383992551943680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1766991720366592 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13406443035136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27560793 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1770040793103932 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20148564 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚397474357548544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚11549 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1756654923627088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47709646 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114243886 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114243886 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207685420 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1780398163401728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93451679 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13481912971264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1780398163401728 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 $ _timestamp‚1.784517317329645e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1780397804938240 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1766991720366592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114233743 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13481805604864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292FqE^¢Ö%„P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13406101611008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚397474483527168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1766991703327232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13406443035136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93451679 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1770041150704566 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13385869476844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27561082 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207695565 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚383992551943680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1766991720366592 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13406443035136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27560793 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1770040793103932 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20148564 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚397474357548544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚11549 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1756654923627088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47709646 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114243886 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114243886 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207685420 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1780398163401728 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93451679 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13481912971264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1780398163401728 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 $ _timestamp‚1.784517317329645e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1780397804938240 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1766991720366592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114233743 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13481805604864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1780397804938240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344µ9ÊòB4×%/ _wandbruntime‚10491  _runtime‚10491Ç‘ÊB'Ø%‹N EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20150295 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27563367 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13407553141248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚397508648350720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚950 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13387318426482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207702869 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207702869 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚11549 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93459472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13386977865096 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1767139977829376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93459472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚384025732213760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114243399 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13482916136960 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1767139994868736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47713665 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207713028 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207713028 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13407553141248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1780547189539328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1770189332159656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93459470 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47713390 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚397508780613632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13407211709952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845173179383488e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1756802354294560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20150298 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114243399 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114253556 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13483029787648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1770189689760402 _runtime‚10491.514081704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ÔõŒ'Ù%N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207713028 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207713028 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13407553141248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1780547189539328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1770189332159656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93459470 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47713390 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚397508780613632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13407211709952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845173179383488e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1756802354294560 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20150298 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114243399 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114253556 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13483029787648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1770189689760402 _runtime‚10491.514081704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20150295 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27563367 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13407553141248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚397508648350720 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚950 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13387318426482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207702869 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207702869 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚11549 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93459472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13386977865096 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1767139977829376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93459472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚384025732213760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114243399 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13482916136960 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1767139994868736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47713665¶¹x—ñ4Ú%/ _wandbruntime‚10491  _runtime‚10491kÕà Û%† 0loss/FRt2/assignment_loss‚0.6594367500403516 3loss/FBt2/assignment_loss‚0.0005562281294480269 /loss/FBt2/detection_loss‚0.5900318432803743 /loss/FRt2/detection_loss‚1.4590985090103294 1loss/SRqqt2/detection_loss‚0.6403063888490579 0loss/FRt1/assignment_loss‚1.5850454034317885 2loss/SRqqt1/assignment_loss‚0.4837558708024331 /loss/FRt1/detection_loss‚0.5007152318610533 1loss/SRqqt1/detection_loss‚0.6682878343365284 0loss/FBt1/detection_loss‚0.40790498472947195 $ _timestamp‚1.784517317941166e+09 _step‚951 2loss/SRqqt2/assignment_loss‚0.5124608190267296 1loss/FBt1/assignment_loss‚0.00372357670268964 trainer/global_step‚11549 _runtime‚10491.516470718 %loss/total_loss‚7.511323440200256 epoch‚28W‚%0Ü%‹ %loss/total_loss‚7.511323440200256 epoch‚28 0loss/FRt2/assignment_loss‚0.6594367500403516 3loss/FBt2/assignment_loss‚0.0005562281294480269 /loss/FBt2/detection_loss‚0.5900318432803743 /loss/FRt2/detection_loss‚1.4590985090103294 1loss/SRqqt2/detection_loss‚0.6403063888490579 0loss/FRt1/assignment_loss‚1.5850454034317885 2loss/SRqqt1/assignment_loss‚0.4837558708024331 /loss/FRt1/detection_loss‚0.5007152318610533 1loss/SRqqt1/detection_loss‚0.6682878343365284 0loss/FBt1/detection_loss‚0.40790498472947195 $ _timestamp‚1.784517317941166e+09 _step‚951 2loss/SRqqt2/assignment_loss‚0.5124608190267296 1loss/FBt1/assignment_loss‚0.00372357670268964 trainer/global_step‚11549 _runtime‚10491.516470718·aAõpÝ%‚(:å Î¥öÒÚ‘™Ë disk.nvme1n1.in‚ 229.87109375# disk.dm-0.out‚83572.84326171875 disk.rbd0.out‚0$ memory_percent‚96.98667526245117 proc.cpu.threads‚17 network.sent‚ 1.8384538e+07) disk./.usagePercent‚76.16713038874533& disk.nvme0c0n1.out‚66048.708984375" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-1.in‚0# disk.dm-1.out‚13415.89599609375& proc.memory.availableMB‚ 493.703125" disk.nvme0c0n1.in‚ 295.87890625$ disk.nvme0n1.out‚66048.708984375% disk.nvme1c1n1.out‚32118.64453125# disk.nvme1n1.out‚32118.64453125 disk.md0.out‚ 567.45703125 disk.dm-0.in‚525.75 disk.rbd0.in‚0) proc.memory.percent‚18.45989227294922 network.recv‚ 2.751969e+06$ disk./.usageGB‚1169.355869293213 disk.nvme0n1.in‚ 295.87890625 proc.memory.rssMB‚ 3024.46875 cpu‚25.68580728579884«ZHžÞ%‚(:“ Î¥öÒÿˆŒÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚259.806' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.60199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–¬íwwß%‚(:ì Ý¥öÒºùÉË proc.memory.rssMB‚ 3024.46875 cpu‚25.686062928598197& disk.nvme1c1n1.out‚32119.353515625 network.sent‚ 1.8446958e+07% disk./.usageGB‚1169.3559608459473' disk.nvme0c0n1.out‚66050.1318359375% disk.nvme0n1.out‚66050.1318359375 disk.md0.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚76.16713635212162 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 295.87890625$ disk.nvme1n1.out‚32119.353515625 disk.md0.out‚ 567.9609375# disk.dm-0.out‚83573.78759765625$ memory_percent‚96.98665142059326" disk.nvme1c1n1.in‚ 229.87109375 network.recv‚ 2.756311e+06 disk.dm-1.in‚0# disk.dm-1.out‚13416.01708984375" disk.nvme0c0n1.in‚ 295.87890625 disk.nvme1n1.in‚ 229.87109375( proc.memory.availableMB‚ 493.70703125) proc.memory.percent‚18.45989227294922 disk.dm-0.in‚525.75J Çà%‚(:’ Ý¥öÒ¯õÌ gpu.0.gpu‚19.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚284.545' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.84833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eš˜&vá%‚(:í ì¥öÒ„ˆ¬Ë disk.md0.out‚ 568.18359375 network.sent‚ 1.8450786e+07 network.recv‚ 2.759381e+0,S ¥6 disk.nvme0n1.in‚ 295.87890625% disk.nvme0n1.out‚66057.2255859375' disk.nvme1c1n1.out‚32120.1240234375# disk.dm-0.out‚83581.08154296875$ memory_percent‚96.98665142059326) disk./.usagePercent‚76.13621525220152' disk.nvme0c0n1.out‚66057.2255859375 disk.dm-1.in‚0) proc.memory.percent‚18.45989227294922 cpu‚25.6861787647474 proc.cpu.threads‚17 disk.md0.in‚0% disk./.usageGB‚1168.8812446594238" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚525.75# disk.dm-1.out‚13416.08154296875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 295.87890625% disk.nvme1n1.out‚32120.1240234375 disk.rbd0.out‚0( proc.memory.availableMB‚ 493.70703125 proc.memory.rssMB‚ 3024.46875·-žâ%‚(:“ ì¥öÒÉ€Ì gpu.0.gpu‚85.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚279.832' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.27733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±M.4ã%/ _wandbruntime‚10535  _runtime‚10535ZKøËä%Å ' lr-AdamW/pg1‚0.00023892434388744218 ' lr-AdamW/pg2‚0.00023892434388744218 trainer/global_step‚11599 % _timestamp‚1.7845173615429955e+09 _runtime‚10535.120010361 _step‚952zô@Ðå%Ê ' lr-AdamW/pg2‚0.00023892434388744218 trainer/global_step‚11599 % _timestamp‚1.7845173615429955e+09 _runtime‚10535.120010361 _step‚952 ' lr-AdamW/pg1‚0.00023892434388744218¸ì| T4æ%/ _wandbruntime‚10535  _runtime‚10535g$ã²(ç%€P UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93841329 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚11599 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208569867 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _step‚953 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114718395 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13441693251526 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13462012136960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93841329 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114728538 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1787867099147264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114718395 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13461670712832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13461670712832 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13441352690252 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1787867457610752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1774405445473792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13537691129856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20235261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1787867099147264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92686336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208559722 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13462012136960 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114728538 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845173615435913e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1774405445473792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111298560 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1774405428434432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20235264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27675874 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚10535.120072921 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399189367876096 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1764026474040048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚385651565447680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1777467809690940 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1787867457610752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399189256577536 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1764026457000688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47911138 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1774405428434432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27675578 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1777468167291574—d4 (è%…P D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13441352690252 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1787867457610752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1774405445473792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13537691129856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20235261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1787867099147264 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92686336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208559722 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13462012136960 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114728538 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845173615435913e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1774405445473792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111298560 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1774405428434432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20235264 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27675874 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚10535.120072921 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399189367876096 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1764026474040048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚385651565447680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1777467809690940 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1787867457610752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399189256577536 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1764026457000688 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47911138 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1774405428434432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27675578 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1777468167291574 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93841329 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚11599 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208569867 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _step‚953 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114718395 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13441693251526 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13462012136960 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93841329 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114728538 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1787867099147264 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114718395 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13461670712832 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13461670712832 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚10535.726962794 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93849122 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1777616348746664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114728051 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27677896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208577171 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1777616706347410 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13463122243072 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13462780811776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1774553702936576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13538903278080 trainer/global_step‚11599 % _timestamp‚1.7845173621512277e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114728051 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27678196 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1764173904707520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13462780811776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208577171 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47914891 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399223667608064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13442801639890 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13442461078504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20236998 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114738208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1774553702936576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1774553719975936 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1788016842219008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385684745717760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208587330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20236995 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1764173887668160 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399223550025216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13538804307456å» Ü»ë%N UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1788016842219008 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114738208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/overÒî´FZ%size_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20236998 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114738208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1774553702936576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1774553719975936 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1788016842219008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385684745717760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208587330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20236995 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1764173887668160 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399223550025216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13538804307456 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚10535.726962794 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93849122 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1777616348746664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114728051 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27677896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208577171 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1777616706347410 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13463122243072 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13462780811776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1774553702936576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13538903278080 trainer/global_step‚11599 % _timestamp‚1.7845173621512277e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114728051 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27678196 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1764173904707520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13462780811776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208577171 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47914891 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399223667608064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13442801639890 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13442461078504 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 Q&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4720524847507477 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 +particle/sensitivity‚0.6521479487419128 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2504751980304718 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7349086999893188 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7074204087257385 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6378747224807739 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7202796339988708 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3245067894458771 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.287888765335083 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14473171532154083 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7798991203308105 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 /particle/accuracy_4_of_4‚0.9496787786483765 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 (particle/accuracy‚0.8567515015602112 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880179166793823 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8120748400688171 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.342477947473526 particle/accuracy_4_of_5‚1 trainer/global_step‚11599 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20284219086170197 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6556364893913269 particle/accuracy_2_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9124957323074341 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7007697224617004 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_4‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02607574127614498 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963566660881042 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8"Purity/2FRt*SRqqt*FBt/event_purity‚0.074837826192379 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7731207609176636 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5620056986808777 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1282283067703247 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5975918769836426 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27521878480911255 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983494877815247 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17469871044158936 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01585601270198822 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9372338652610779 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 (loss/val_total_loss‚7.61927557491306 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6378747224807739 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8918935060501099 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3019946813583374 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.31580081582069397 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6805463433265686 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6616128087043762 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6566541790962219 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.722244381904602 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6704038977622986 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2734358310699463 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8581452369689941 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 epoch‚28 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9961692094802856 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04596278816461563 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7182427644729614 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6297712326049805 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9938672184944153 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8955782055854797 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.37224799394607544 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.484375 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6871111989021301 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7916509509086609 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7573790550231934 +jet/accuracy_3_of_4‚0.09568284451961517 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.33579912781715393 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7665637731552124 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6359527707099915 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5764540433883667 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0662548691034317 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.801782488822937 +jet/accuracy_3_of_3‚0.07661568373441696 particle/accuracy_1_of_2‚1 _runtime‚10551.487284325 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998860239982605 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8924273252487183 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6651047468185425 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6653087735176086 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5458008646965027 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.700445294380188 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.20000000298023224 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5600000023841858 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚1 % _timestamp‚1.7845173779112556e+09 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38368380069732666 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6815430521965027 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7263709306716919 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0569717101752758 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8581452369689941 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24578210711479187 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3242678940296173 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030125590041279793 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3772650957107544 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27629321813583374 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6535831093788147 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7962962985038757 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02588861621916294 *jet/accuracy_2_of_4‚0.7956969141960144 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6893149018287659 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3346746861934662 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7962962985038757 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1891990751028061 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8683798909187317 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.659995436668396 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7862351536750793 jet/accuracy_3_of_5‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.32420775294303894 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8884731531143188 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6750292778015137 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 _step‚956 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45584627985954285 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.27718666195869446 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8896258473396301 *jet/accuracy_2_of_2‚0.1724550575017929 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25900834798812866 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6377254128456116 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7validation_average_jet_accuracy‚0.40090033411979675 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5162258148193359 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3242678940296173 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7805860638618469 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43518516421318054 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9551908373832703 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918821454048157 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6781213879585266zäŠ4“ó%»z 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2504751980304718 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7349086999893188 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7074204087257385 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6378747224807739 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7202796339988708 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3245067894458771 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.287888765335083 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.14473171532154083 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7798991203308105 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 /particle/accuracy_4_of_4‚0.9496787786483765 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 (particle/accuracy‚0.8567515015602112 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9880179166793823 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8120748400688171 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.342477947473526 particle/accuracy_4_of_5‚1 trainer/global_step‚11599 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.20284219086170197 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6556364893913269 particle/accuracy_2_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9124957323074341 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7007697224617004 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_4‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02607574127614498 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963566660881042 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8"Purity/2FRt*SRqqt*FBt/event_purity‚0.074837826192379 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7731207609176636 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.5620056986808777 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1282283067703247 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5975918769836426 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27521878480911255 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983494877815247 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17469871044158936 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.01585601270198822 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9372338652610779 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 (loss/val_total_loss‚7.61927557491306 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6378747224807739 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8918935060501099 9"Purity/1FRt1SRqqt1FBt/event_[&ˆ®+purity‚0.3019946813583374 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.31580081582069397 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6805463433265686 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6616128087043762 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6566541790962219 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.722244381904602 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6704038977622986 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2734358310699463 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8581452369689941 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 epoch‚28 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9961692094802856 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04596278816461563 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7182427644729614 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6297712326049805 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9938672184944153 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8955782055854797 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.37224799394607544 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.484375 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6871111989021301 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7916509509086609 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7573790550231934 +jet/accuracy_3_of_4‚0.09568284451961517 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.33579912781715393 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7665637731552124 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6359527707099915 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5764540433883667 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0662548691034317 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.801782488822937 +jet/accuracy_3_of_3‚0.07661568373441696 particle/accuracy_1_of_2‚1 _runtime‚10551.487284325 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998860239982605 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8924273252487183 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6651047468185425 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6653087735176086 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.5458008646965027 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.700445294380188 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.20000000298023224 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5600000023841858 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚1 % _timestamp‚1.7845173779112556e+09 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.38368380069732666 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6815430521965027 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.7263709306716919 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0569717101752758 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8581452369689941 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.24578210711479187 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3242678940296173 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030125590041279793 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3772650957107544 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27629321813583374 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6535831093788147 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7962962985038757 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02588861621916294 *jet/accuracy_2_of_4‚0.7956969141960144 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6893149018287659 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3346746861934662 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7962962985038757 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1891990751028061 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8683798909187317 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.659995436668396 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7862351536750793 jet/accuracy_3_of_5‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.32420775294303894 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8884731531143188 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6750292778015137 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 _step‚956 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45584627985954285 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.27718666195869446 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8896258473396301 *jet/accuracy_2_of_2‚0.1724550575017929 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25900834798812866 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6377254128456116 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7validation_average_jet_accuracy‚0.40090033411979675 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5162258148193359 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3242678940296173 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7805860638618469 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43518516421318054 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9551908373832703 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918821454048157 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6781213879585266 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.286344975233078 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18865248560905457 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7106148600578308 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.01585601270198822 particle/accuracy_1_of_3‚1 +particle/specificity‚0.9127074480056763 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38077953457832336 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6329390406608582 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9896021485328674 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9889349341392517 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4690146744251251 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6956343650817871 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.36330747604370117 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9702250361442566 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8995842933654785 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9955945611000061 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5882418155670166 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.33874374628067017 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7265625596046448 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.25454410910606384 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24509234726428986 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.286235511302948 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 particle/accuracy_3_of_5‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.33488085865974426 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.029824137687683105 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_1‚1 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5819940567016602 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4720524847507477 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 +particle/sensitivity‚0.6521479487419128¼Ô¾•ivô%‚(:ë Š¦öÒÛ„ïÍ" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚95.49479484558105' proc.memory.availableMB‚ 738.1328125" proc.memory.rssMB‚ 3025.7890625) proc.memory.percent‚18.46795082092285 cpu‚25.68925443351853 proc.cpu.threads‚17 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 295.87890625( disk.nvme0c0n1.out‚66071.74560546875 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚525.75! disk.dm-0.out‚83597.900390625# disk.dm-1.out‚13661.76220703125 disk.nvme0n1.in‚ 295.87890625& disk.nvme0n1.out‚66071.74560546875& disk.nvme1c1n1.out‚32369.505859375 network.recv‚ 2.767169e+06) disk./.usagePercent‚76.13624034807671% disk./.usageGB‚1168.8816299438477$ disk.nvme1n1.out‚32369.505859375 disk.md0.in‚0 disk.md0.out‚ 568.8359375 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 1.851922e+07,«žõ%‚(:“ Š¦öÒ–ŠâË gpu.0.gpu‚52.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚283.057' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.35233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡Øxö%‚(:í ™¦öÒÏôÐË network.sent‚ 1.8574864e+07 disk.dm-0.in‚525.75 disk.rbd0.out‚0) proc.memory.percent‚18.46795082092285 cpu‚25.689068562169812 network.recv‚ 2.77137e+06 disk.dm-1.in‚0# disk.dm-1.out‚13697.83251953125 disk.nvme0n1.in‚ 295.87890625& disk.nvme0n1.out‚66073.79638671875" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 569.27734375 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 295.87890625 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚32406.111328125 disk.md0.in‚0( proc.memory.availableMB‚ 737.38671875" proc.memory.rssMB‚ 3025.7890625) disk./.usagePercent‚76.14361356590183% disk./.usageGB‚1168.9948272705078! disk.dm-0.out‚83599.525390625( disk.nvme0c0n1.out‚66073.79638671875& disk.nvme1c1n1.out‚32406.111328125# memory_percent‚95.4993486404419 proc.cpu.threads‚17 ~·“÷%‚(:ˆ ™¦öÒ¤½©Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.866' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.622 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëm›vsø%‚(:è ¨¦öÒ…ŒàË disk.dm-0.out‚84105.17578125# disk.dm-1.out‚13907.39306640625" disk.nvme0c0n1.in‚ 295.87890625" proc.memory.rssMB‚ 3025.7890625) disk./.usagePercent‚76.16847438467498& disk.nvme0n1.out‚66523.84130859375 disk.nvme1n1.in‚ 229.87109375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚66523.84130859375$ memory_percent‚95.50013542175293' proc.memory.availableMB‚ 737.2578125 proc.cpu.threads‚17% disk./.usageGB‚1169.3765029907227% disk.nvme1c1n1.out‚32672.08984375# disk.nvme1n1.out‚32672.08984375 disk.md0.in‚0 disk.rbd0.out‚0 cpu‚25.689106990202937 network.sent‚ 1.8581598e+07 disk.dm-0.in‚525.75 disk.dm-1.in‚0) proc.memory.percent‚18.46795082092285 network.recv‚ 2.774528e+06 disk.nvme0n1.in‚ 295.87890625" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 569.65625÷•Â:ù%‚(:’ ¨¦öÒ¹­Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.629' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.209666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ZU4ú%/ _wandbruntime‚10595  _runtime‚10595nCЯÈû% _runtime‚10595.278747518 _step‚957 & lr-AdamW/pg1‚0.0002365258707314755 & lr-AdamW/pg2‚0.0002365258707314755 trainer/global_step‚11649 $ _timestamp‚1.784517421703074e+09ŠàlÍü%Ç & lr-AdamW/pg2‚0.0002365258707314755 trainer/global_step‚11649 $ _timestamp‚1.784517421703074e+09 _runtime‚10595.278747518 _step‚957 & lr-AdamW/pg1‚0.0002365258707314755½4ëqN4ý%/ _wandbruntime‚10595  _runtime‚10595©u¨(þ%P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115284546 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1797299055070208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209648476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94353785 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1783761497417216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13537557652992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚387333176226304 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1773319246382512 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚10595.280825382 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27824989 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1786836386135074 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27825278 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1773319263421872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13537899077120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚400947417095168 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13517480313836 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1786836743735708 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94353787 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚387333157614080 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115294689 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 _step‚958 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13537899077120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13537557652992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94353785 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1797299413533696 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1783761514456576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1797299055070208 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115284546 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13614259481088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209638331 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1797299413533696 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1783761514456576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20326080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2èÔËÞÄÿ%†P ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1797299413533696 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1783761514456576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1797299055070208 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115284546 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13614259481088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209638331 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1797299413533696 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1783761514456576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20326080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115284546 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1797299055070208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209648476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94353785 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1783761497417216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13537557652992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚387333176226304 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1773319246382512 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚10595.280825382 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27824989 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1786836386135074 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27825278 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1773319263421872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13537899077120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚400947417095168 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13517480313836 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1786836743735708 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94353787 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚387333157614080 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115294689 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 _step‚958 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13537899077120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13537557652992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94353785 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452¾púÃ4€&/ _wandbruntime‚10595  _runtime‚105956b˜'&‹N QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13518248140814 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13539009183232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _step‚959 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209665939 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1783909788958720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115304359 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚11649 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1797448439671296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48154981 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1797448798141952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13615485498880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94361580 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13538667751936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚387366337884160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13615373945344 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1783909788958720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94361580 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13538667751936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209665939 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13539009183232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚400981711829504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1783909771919360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚400981841995264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115304359 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209655780 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115294202 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1773466694089344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1797448798141952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48155275 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94361578 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20327814 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13518588702200 % _timestamp‚1.7845174223285265e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27827170 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1773466677049984 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1783909771919360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚387366356496384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94361578 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0®³VI'‚&N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209655780 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20327814 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13518588702200 % _timestamp‚1.7845174223285265e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27827170 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1773466677049984 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1783909771919360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚387366356496384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94361578 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13518248140814 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13539009183232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _step‚959 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209665939 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1783909788958720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115304359 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚11649 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1797448439671296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48154981 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1797448798141952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13615485498880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94361580 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13538667751936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚387366337884160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13615373945344 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1783909788958720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94361580 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13538667751936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209665939 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13539009183232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚400981711829504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1783909771919360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚400981841995264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115304359 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209655780 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115294202 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1773466694089344 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1797448798141952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48155275 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94361578 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13593468178944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13670264719872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13670163644928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115779341 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1791175222524416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_ba‰ÙãÄtch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13572963527244 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1804768707742720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402662334763008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1794263760322716 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1780690779756112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210522778 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845174664822962e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210512633 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388992189730304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48352499 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27939996 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94743437 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115779341 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94743435 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1804768707742720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388992171118080 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚10640.057925738 trainer/global_step‚11699 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1794263402722082 _step‚962 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1804768349279232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20412780 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115769198 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1791175239563776 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94743435 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1780690796795472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1804768349279232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20412777 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13593126754816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94743437 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13593126754816 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚10640.057925738 trainer/global_step‚11699 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1794263402722082 _step‚962 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1804768349279232 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20412780 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115769198 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1791175239563776 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94743435 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1780690796795472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1804768349279232 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20412777 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13593126754816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94743437 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13593126754816 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13593468178944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13670264719872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13670163644928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115779341 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1791175222524416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13572963527244 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1804768707742720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402662334763008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1794263760322716 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1780690779756112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210522778 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845174664822962e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210512633 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388992189730304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48352499 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27939996 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94743437ÂØËŒW4’&/ _wandbruntime‚10640  _runtime‚10640ÊÜýn'“&‹N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115789011 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13573731354222 trainer/global_step‚11699 _runtime‚10640.683925907 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94751230 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115789011 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚402696754076160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 % _timestamp‚1.7845174671081002e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚389025370000384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210530082 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94751230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20414514 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1780838227462944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210540241 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94751228 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48356810 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚389025351388160 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402696626007552 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1794411941777806 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1791323497026560 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13594578285056 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210540241 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1794412299378552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94751228 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13594236853760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210530082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20414511 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27942296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13671384075776 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1780838210423584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚306 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27941993 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚963 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115778854 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13594236853760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13671274619392 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1804917733880320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1791323497026560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13594578285056 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48356504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0œ"VÈ”&N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚306 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27941993 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚963 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115778854 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13594236853760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13671274619392 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1804917733880320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1791323497026560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13594578285056 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48356504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115789011 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13573731354222 trainer/global_step‚11699 _runtime‚10640.683925907 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94751230 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115789011 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚402696754076160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 % _timestamp‚1.7845174671081002e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚389025370000384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210530082 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94751230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20414514 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1780838227462944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210540241 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94751228 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48356810 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚389025351388160 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402696626007552 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1794411941777806 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1791323497026560 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13594578285056 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210540241 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1794412299378552 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94751228 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13594236853760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210530082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20414511 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27942296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13671384075776 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1780838210423584Ãâ§L4•&/ _wandbruntime‚10640  _runtime‚10640²: –&† 2loss/SRqqt2/assignment_loss‚0.5230042470995016 1loss/SRqqt2/detection_loss‚0.6555044770486986 /loss/FBt1/detection_loss‚0.4795694215507764 _runtime‚10640.685001719 _step‚964 0loss/FRt1/assignment_loss‚1.6167602483047185 0loss/FRt2/assignment_loss‚0.6817085735499859 3loss/FBt1/assignment_loss‚0.0030117344978510432 /loss/FRt2/detection_loss‚1.6209105839789846 trainer/global_step‚11699 1loss/FBt2/assignment_loss‚0.02428328598408669 /loss/FRt1/detection_loss‚0.4664656490690571 1loss/SRqqt1/detection_loss‚0.7461186062489666 % _timestamp‚1.7845174671096115e+09 /loss/FBt2/detection_loss‚0.8066851888464666 %loss/total_loss‚8.113067409553548 epoch‚29 2loss/SRqqt1/assignment_loss‚0.4890453933744533%³+f—&‹ 0loss/FRt2/assignment_loss‚0.6817085735499859 3loss/FBt1/assignment_loss‚0.0030117344978510432 /loss/FRt2/detection_loss‚1.6209105839789846 trainer/global_step‚11699 1loss/FBt2/assignment_loss‚0.02428328598408669 /loss/FRt1/detection_loss‚0.4664656490690571 1loss/SRqqt1/detection_loss‚0.7461186062489666 % _timestamp‚1.7845174671096115e+09 /loss/FBt2/detection_loss‚0.8066851888464666 %loss/total_loss‚8.113067409553548 epoch‚29 2loss/SRqqt1/assignment_loss‚0.4890453933744533 2loss/SRqqt2/assignment_loss‚0.5230042470995016 1loss/SRqqt2/detection_loss‚0.6555044770486986 /loss/FBt1/detection_loss‚0.4795694215507764 _runtime‚10640.685001719 _step‚964 0loss/FRt1/assignment_loss‚1.6167602483047185Ä…8z®y˜&‚(:î ä¦öÒËøË& disk.nvme0c0n1.out‚66696.751953125 disk.nvme0n1.in‚ 295.87890625# disk.dm-0.out‚84279.22802734375) proc.memory.percent‚18.46795082092285 disk.rbd0.in‚0$ disk.nvme0n1.out‚66696.751953125 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 571.48046875 disk.dm-0.in‚525.75$ memory_percent‚95.50273418426514 cpu‚25.68972981569558 proc.cpu.threads‚17 network.recv‚ 2.788928e+06) disk./.usagePercent‚76.16005807294847" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0" proc.memory.rssMB‚ 3025.7890625 network.sent‚ 1.8668775e+07% disk./.usageGB‚1169.2472915649414" disk.nvme0c0n1.in‚ 295.87890625% disk.nvme1n1.out‚32677.6474609375( proc.memory.availableMB‚ 736.83203125' disk.nvme1c1n1.out‚32677.6474609375 disk.dm-1.in‚0# disk.dm-1.out‚13907.90478515625 disk.rbd0.out‚0H{ -œ™&‚(:‘ ä¦öÒ§îºÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.085' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.02833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c£Šäjš&‚(:ß ó¦öÒÓ‚ËË" disk.nvme1c1n1.in‚ 229.87109375" disk.nvme0c0n1.in‚ 295.87890625( disk.nvme0c0n1.out‚66803.39599609375 disk.nvme0n1.in‚ 295.87890625 disk.nvme1c1n1.out‚ 32715.375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-0.in‚525.75 disk.dm-1.in‚0# disk.dm-1.out‚13907.97509765625 disk.rbd0.out‚0$ disk./.usageGB‚1169.484661102295 disk.nvme1n1.out‚ 32715.375$ memory_percent‚95.50275802612305& proc.memory.availableMB‚ 736.828125) proc.memory.percent‚18.46795082092285 proc.cpu.threads‚17& disk.nvme0n1.out‚66803.39599609375 disk.md0.out‚ 572.13671875 disk.dm-0.out‚84422.13671875 disk.rbd0.in‚0" proc.memory.rssMB‚ 3025.7890625 cpu‚25.689956697190887 network.sent‚ 1.8718314e+07 network.recv‚ 2.793215e+06) disk./.usagePercent‚76.17551936833055'6x‘›&‚(:† ó¦öÒúöŸÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.195' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.065 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\¦š{œ&‚(:𠂧öÒ´‘ÞË disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0" disk.dm-1.out‚13908.1630859375 disk.rbd0.in‚0 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0$ memory_percent‚95.50275802612305 network.sent‚ 1.8725074e+07" disk.nvme0c0n1.in‚ 295.87890625% disk.nvme0n1.out‚66885.9833984375" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚32745.43408203125 disk.md0.out‚ 572.55859375& proc.memory.availableMB‚ 736.828125 cpu‚25.690016770660023) disk./.usagePercent‚76.17552334391475' disk.nvme0c0n1.out‚66885.9833984375 disk.nvme0n1.in‚ 295.87890625( disk.nvme1c1n1.out‚32745.43408203125 disk.dm-0.in‚525.75# disk.dm-0.out‚84533.68896484375" proc.memory.rssMB‚ 3025.7890625) proc.memory.percent‚18.46795082092285 network.recv‚ 2.796376e+06% disk./.usageGB‚1169.4847221374512¶„ ž&‚(:“ ‚§öÒþ˜¢Ì gpu.0.gpu‚57.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚288.259' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.08633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž£—…4ž&/ _wandbruntime‚10683  _runtime‚10683œ 8ÊŸ&Ä trainer/global_step‚11749 $ _timestamp‚1.784517510422972e+09 _runtime‚10683.998573013 _step‚965 ' lr-AdamW/pg1‚0.00023174270459413066 ' lr-AdamW/pg2‚0.00023174270459413066¥í}½Ï &É ' lr-AdamW/pg1‚0.00023174270459413066 ' lr-AdamW/pg2‚0.00023174270459413066 trainer/global_step‚11749 $ _timestamp‚1.784517510422972e+09 _runtime‚10683.998573013 _step‚965ÅtòâG4¡&/ _wandbruntime‚10683  _runtime‚10683öþCS(¢&ÿO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 # _timestamp‚1.78451751042389e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20499477 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1812237643488256 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116263993 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13649037280768 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116263993 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1798588964670976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _step‚966 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95133085 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1798588947631616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13648695856640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1798588964670976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28054437 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1788062330169072 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1798588947631616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 _runtime‚10683.999351874 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13648695856640 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1801690419309090 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48553636 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48553917 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390651184622080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13725999944704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211397080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116253850 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1788062313129712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13628106179378 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1812238001951744 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13649037280768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28054159 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95133085 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95133087 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1812237643488256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20499480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404377184566784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1801690776909724 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1812238001951744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390651203234304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚404377306351104 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48553917 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390651184622080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13725999944704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211397080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116253850 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1788062313129712 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13628106179378 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1812238001951744 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13649037280768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28054159 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95133085 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95133087 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1812237643488256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20499480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404377184566784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1801690776909724 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1812238001951744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390651203234304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚404377306351104 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1812237643488256 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116263993 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13649037280768 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116263993 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1798588964670976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _step‚966 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95133085 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ÆÒÐFä4¤&/ _wandbruntime‚10684  _runtime‚10684 ßzv¥&‹N XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1812387386560000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚390684364892160 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1801838958364814 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404411477818368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1812387386560000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_ençFõ›!d/reserved_bytes.all.freed‚0 trainer/global_step‚11749 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1812387028089344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211404384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95140880 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13629555129016 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 % _timestamp‚1.7845175110332046e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1798737239173120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211404384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1798737222133760 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48557587 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20501211 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1801839315965560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚967 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚10684.608967348 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211414543 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13727220285440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95140878 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28056656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13727112926208 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13629214567630 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95140878 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1798737239173120 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1798737222133760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116273663 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211414543 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13650147386880 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116263506 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13650147386880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13649805955584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116273663 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95140878 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1798737239173120 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1798737222133760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116273663 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211414543 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13650147386880 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116263506 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13650147386880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13649805955584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116273663 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11749 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1812387028089344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211404384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95140880 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13629555129016 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 % _timestamp‚1.7845175110332046e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1798737239173120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211404384 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1798737222133760 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48557587 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20501211 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1801839315965560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚967 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚10684.608967348 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211414543 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13727220285440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95140878 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28056656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13727112926208 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13629214567630 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184Ç&Âta4§&/ _wandbruntime‚10684  _runtime‚10684J²«Ã ¨&† trainer/global_step‚11749 _runtime‚10684.611269172 0loss/FRt2/assignment_loss‚0.8023876359808111 3loss/FBt2/assignment_loss‚0.0022845593274349874 1loss/SRqqt2/detection_loss‚0.6795797831659643 0loss/FBt1/detection_loss‚0.41554334876705173 /loss/FBt2/detection_loss‚0.5619292363116983 epoch‚29 3loss/SRqqt1/assignment_loss‚0.46121265751743123 1loss/FBt1/assignment_loss‚0.00493852822959418 0loss/FRt1/detection_loss‚0.44808564598861267 0loss/SRqqt1/detection_loss‚0.677876086788564 _step‚968 0loss/FRt1/assignment_loss‚1.5737237460786404 3loss/SRqqt2/assignment_loss‚0.48161346651011694 % _timestamp‚1.7845175110359592e+09 -loss/FRt2/detection_loss‚1.50659197022503 $loss/total_loss‚7.61576666489095ª¼r ©&‹ trainer/global_step‚11749 _runtime‚10684.611269172 0loss/FRt2/assignment_loss‚0.8023876359808111 3loss/FBt2/assignment_loss‚0.0022845593274349874 1loss/SRqqt2/detection_loss‚0.6795797831659643 0loss/FBt1/detection_loss‚0.41554334876705173 /loss/FBt2/detection_loss‚0.5619292363116983 epoch‚29 3loss/SRqqt1/assignment_loss‚0.46121265751743123 1loss/FBt1/assignment_loss‚0.00493852822959418 0loss/FRt1/detection_loss‚0.44808564598861267 0loss/SRqqt1/detection_loss‚0.677876086788564 _step‚968 0loss/FRt1/assignment_loss‚1.5737237460786404 3loss/SRqqt2/assignment_loss‚0.48161346651011694 % _timestamp‚1.7845175110359592e+09 -loss/FRt2/detection_loss‚1.50659197022503 $loss/total_loss‚7.61576666489095È? ‹}ª&‚(:ò ‘§öÒüíñË network.recv‚ 2.799789e+06 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚66899.56884765625& disk.nvme1n1.out‚32748.33251953125" proc.memory.rssMB‚ 3025.7890625 proc.cpu.threads‚17) disk./.usagePercent‚76.16008465966773 disk.nvme0n1.in‚ 295.87890625 disk.md0.in‚0 disk.dm-0.in‚525.75' proc.memory.availableMB‚ 738.1640625) proc.memory.percent‚18.46795082092285 cpu‚25.689633188088855 network.sent‚ 1.8740834e+07% disk./.usageGB‚1169.2476997375488" disk.dm-1.out‚13908.2607421875" disk.nvme0c0n1.in‚ 295.87890625" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚32748.33251953125 disk.md0.out‚ 573.02734375$ memory_percent‚95.49460411071777( disk.nvme0c0n1.out‚66899.56884765625 disk.nvme1n1.in‚ 229.87109375" disk.dm-0.out‚84549.0751953125{[ê)“«&‚(:ˆ ‘§öÒ•´Ì gpu.0.gpu‚95.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚264.054' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚88.018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²Uöt¬&‚(:é  §öÒÓÑÇË network.recv‚ 2.804066e+06% disk./.usageGB‚1169.2483825683594( disk.nvme0c0n1.out‚66901.48779296875& disk.nvme1c1n1.out‚32748.966796875 disk.md0.out‚ 573.4140625 disk.dm-1.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 295.87890625 disk.nvme0n1.in‚ 295.87890625& disk.nvme0n1.out‚66901.48779296875" disk.nvme1c1n1.in‚ 229.87109375$ disk.nvme1n1.out‚32748.966796875) proc.memory.percent‚18.46795082092285 cpu‚25.68927642584567 network.sent‚ 1.879232e+07 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0# disk.dm-0.out‚84550.66943359375 disk.rbd0.in‚0$ memory_percent‚95.49460411071777 proc.cpu.threads‚17) disk./.usagePercent‚76.16012913651585 disk.dm-0.in‚525.75 disk.dm-1.out‚ 13908.3828125' proc.memory.availableMB‚ 738.1640625" proc.memory.rssMB‚ 3025.7890625xÓ«{Ÿ­&‚(:”  §öÒ˜õ„Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚140.184' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚46.727999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BwTåp®&‚(:å ¯§öÒúðÓË" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚573.75$ memory_percent‚95.49460411071777) proc.memory.percent‚18.46795082092285 network.sent‚ 1.879715e+07 network.recv‚ 2.807613e+06) disk./.usagePercent‚76.16013013041189 disk.dm-1.in‚0 disk.dm-1.out‚ 13908.4296875$ disk.nvme0n1.out‚66908.951171875& disk.nvme1n1.out‚32750.82861328125 disk.md0.in‚0 disk.nvme0n1.in‚ 295.87890625( disk.nvme1c1n1.out‚32750.82861328125 disk.nvme1n1.in‚ 229.87109375# disk.dm-0.out‚84559.20849609375' proc.memory.availableMB‚ 738.1640625" proc.memory.rssMB‚ 3025.7890625 cpu‚25.688918523216334 disk.rbd0.out‚0 disk.dm-0.in‚525.75 proc.cpu.threads‚17% disk./.usageGB‚1169.2483978271484 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 295.87890625& disk.nvme0c0n1.out‚66908.951171875˜xë➯&‚(:“ ¯§öÒÚìŽÌ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚268.219' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.40633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ô,’4°&/ _wandbruntime‚10726  _runtime‚10726U Y‹É±&à trainer/global_step‚11799 % _timestamp‚1.7845175527812257e+09 _runtime‚10726.357434035 _step‚969 & lr-AdamW/pg1‚0.0002293583188265754 & lr-AdamW/pg2‚0.0002293583188265754àñ¼Î²&È & lr-AdamW/pg2‚0.0002293583188265754 trainer/global_step‚11799 % _timestamp‚1.7845175527812257e+09 _runtime‚10726.357434035 _step‚969 & lr-AdamW/pg1‚0.0002293583188265754É«æ4³&/ _wandbruntime‚10726  _runtime‚10726г^j^´&P aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1795433863542672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13683929954060 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95522735 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1806002689778176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1819706937697280 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1819707296160768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95522737 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1809117435896098 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116748645 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116738502 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13781831027712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚406092029153792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48754737 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360JÄW (µ&†P TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1819706937697280 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1819707296160768 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95522737 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1809117435896098 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116748645 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116738502 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13781831027712 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚406092029153792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48754737 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1795433863542672 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13683929954060 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95522735 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1806002689778176Êö×9C4¶&/ _wandbruntime‚10726  _runtime‚10726žË«'·&‹N H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1806150947240960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392343397008384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13705716488704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95530528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13782949006336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13684697781038 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116748158 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1806150964280320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1795581277170784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95530530 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1819856322298368 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212288845 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13783050074112 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13705716488704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48759110 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1809266332552568 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13705375057408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1806150947240960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95530528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚11799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28170914 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚971 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1819856680769024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 % _timestamp‚1.7845175533894753e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1806150964280320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20587911 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚406126447082496 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13685038342424 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212288845 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116758315 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212278686 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212278686 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚406126327402496 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1809265974951822 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48758825 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392343378396160 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1795581294210144 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1819856680769024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212288845 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13783050074112 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13705716488704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48759110 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1809266332552568 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13705375057408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1806150947240960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95530528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚11799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28170914 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚971 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1819856680769024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 % _timestamp‚1.7845175533894753e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1806150964280320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20587911 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚406126447082496 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13685038342424 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212288845 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116758315 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212278686 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212278686 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚406126327402496 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1809265974951822 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48758825 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392343378396160 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1795581294210144 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1819856680769024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392343397008384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13705716488704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95530528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13782949006336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13684697781038 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116748158 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1806150964280320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1795581277170784 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232Ë„P¿4¹&/ _wandbruntime‚10726  _runtime‚10726Ò}¥f‹º&ˆ _runtime‚10726.968948452 _step‚972 0loss/FRt1/assignment_loss‚1.5516401920171403 0loss/FBt1/detection_loss‚0.44056¢!çƒ876566003456 /loss/FBt2/detection_loss‚0.5550720208533564 2loss/SRqqt1/assignment_loss‚0.4313763080462263 2loss/FBt1/assignment_loss‚0.007832687117493055 3loss/FBt2/assignment_loss‚0.0019851875597110763 0loss/FRt1/detection_loss‚0.45277071564831134 .loss/FRt2/detection_loss‚1.471647058939901 1loss/SRqqt2/detection_loss‚0.6386529465976615 %loss/total_loss‚7.535795094538429 trainer/global_step‚11799 0loss/FRt2/assignment_loss‚0.7939768645305123 2loss/SRqqt2/assignment_loss‚0.5307296746464097 1loss/SRqqt1/detection_loss‚0.6595426729216706 epoch‚29 % _timestamp‚1.7845175533936157e+09çX²»& 2loss/SRqqt1/assignment_loss‚0.4313763080462263 2loss/FBt1/assignment_loss‚0.007832687117493055 3loss/FBt2/assignment_loss‚0.0019851875597110763 0loss/FRt1/detection_loss‚0.45277071564831134 .loss/FRt2/detection_loss‚1.471647058939901 1loss/SRqqt2/detection_loss‚0.6386529465976615 %loss/total_loss‚7.535795094538429 trainer/global_step‚11799 0loss/FRt2/assignment_loss‚0.7939768645305123 2loss/SRqqt2/assignment_loss‚0.5307296746464097 1loss/SRqqt1/detection_loss‚0.6595426729216706 epoch‚29 % _timestamp‚1.7845175533936157e+09 _runtime‚10726.968948452 _step‚972 0loss/FRt1/assignment_loss‚1.5516401920171403 0loss/FBt1/detection_loss‚0.44056876566003456 /loss/FBt2/detection_loss‚0.5550720208533564ÌÁ¢&`u¼&‚(:ê ¾§öÒã™’Ë disk.rbd0.in‚0' proc.memory.availableMB‚ 738.1015625 disk.dm-0.in‚525.75# disk.dm-1.out‚13908.59814453125 network.sent‚ 1.8800215e+07 network.recv‚ 2.811118e+06& disk.nvme1c1n1.out‚32751.681640625$ disk.nvme1n1.out‚32751.681640625 disk.md0.in‚0" disk.nvme0c0n1.in‚ 295.87890625 disk.md0.out‚ 574.171875# disk.dm-0.out‚84560.41943359375 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚66910.3857421875 disk.nvme0n1.in‚ 295.87890625% disk.nvme0n1.out‚66910.3857421875$ memory_percent‚95.49498558044434( disk./.usagePercent‚76.1603840708519$ disk./.usageGB‚1169.252296447754" disk.nvme1c1n1.in‚ 229.87109375" proc.memory.rssMB‚ 3025.7890625) proc.memory.percent‚18.46795082092285 cpu‚25.688564664714963 proc.cpu.threads‚17 disk.nvme1n1.in‚ 229.87109375E‘Œ“½&‚(:ˆ ¾§öÒ’¹‡Ì gpu.0.gpu‚65.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚272.091' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚90.697 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜*áz¾&‚(:ï ͧöÒØž´Ë" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚66946.5732421875 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 574.703125# disk.dm-0.out‚84669.40673828125( proc.memory.availableMB‚ 491.83203125% disk./.usageGB‚1169.5732612609863' disk.nvme0c0n1.out‚66946.5732421875' disk.nvme1c1n1.out‚32825.7451171875% disk.nvme1n1.out‚32825.7451171875 disk.md0.in‚0 disk.dm-0.in‚525.75 disk.rbd0.out‚0$ memory_percent‚96.99809551239014 network.sent‚ 1.8863627e+07 network.recv‚ 2.815336e+06" disk.nvme0c0n1.in‚ 295.87890625 disk.nvme0n1.in‚ 295.87890625# disk.dm-1.out‚13908.71923828125" proc.memory.rssMB‚ 3025.7890625) proc.memory.percent‚18.46795082092285 cpu‚25.688161492419773) disk./.usagePercent‚76.18129042572782w„¿ô¿&‚(:’ ͧöÒý¡‡Ì gpu.0.gpu‚25.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚122.059' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.68633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05‡4À&/ _wandbruntime‚10769  _runtime‚10769Ë,I´ËÁ&Å _step‚973 ' lr-AdamW/pg1‚0.00022697893555644283 ' lr-AdamW/pg2‚0.00022697893555644283 trainer/global_step‚11849 % _timestamp‚1.7845175955010111e+09 _runtime‚10769.076573196b˜TÆÐÂ&Ê ' lr-AdamW/pg2‚0.00022697893555644283 trainer/global_step‚11849 % _timestamp‚1.7845175955010111e+09 _runtime‚10769.076573196 _step‚973 ' lr-AdamW/pg1‚0.00022697893555644283Íñ'µš4Ã&/ _wandbruntime‚10769  _runtime‚107692¼¨#(Ä&P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117223154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚393969230242304 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1816544810083740 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1816544452483106 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117233297 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1827176590369792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117233297 % _timestamp‚1.7845175955020895e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13760175484416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407806857815040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95912387 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13759834060288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95912387 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1827176231906304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1827176590369792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13759834060288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1813416397846016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48957480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13837646184960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20672877 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95912385 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20672880 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95912385 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1802805379876912 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1813416397846016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213135539 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚407806981696512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1813416414885376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48957198 _runtime‚10769.077675128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213145684 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13837751454208 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13739413167468 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚393969211630080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13760175484416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28284600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚974 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1813416414885376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28284321 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1827176231906304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11849 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584†¤°ò (Å&†P F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117223154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚393969230242304 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1816544810083740 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1816544452483106 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117233297 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1827176590369792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117233297 % _timestamp‚1.7845175955020895e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13760175484416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407806857815040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95912387 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13759834060288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95912387 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1827176231906304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1827176590369792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13759834060288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1813416397846016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48957480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13837646184960 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20672877 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95912385 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20672880 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95912385 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1802805379876912 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1813416397846016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213135539 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚407806981696512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1813416414885376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48957198 _runtime‚10769.077675128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213145684 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13837751454208 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13739413167468 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚393969211630080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13760175484416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28284600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚974 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1813416414885376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28284321 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1827176231906304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11849 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145ά’Î4Æ&/ _wandbruntime‚10769  _runtime‚10769,ßP5ŸÇ&‹N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1802952827583744 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13740521555832 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1813564689387520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20674611 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13838866861568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13760944159232 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1813564672348160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13761285590528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28286604 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13761285590528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95920180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20674614 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95920178 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚975 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394002391900160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117232810 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213163147 trainer/global_step‚11849 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117242967 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13740180994446 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48961502 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 213163147 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95920180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 213152988 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407841277373952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1816693349139576 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13838757405184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚10769.699386502 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1813564689387520 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1813564672348160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213152988 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48961215 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394002410512384 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1802952810544384 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1816692991538830 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7845175961236048e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1827325974978048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28286888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 213163147 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95920180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 213152988 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚407841277373952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1816693349139576 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13838757405184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚10769.699386502 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1813564689387520 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1813564672348160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213152988 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48961215 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394002410512384 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1802952810544384 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1816692991538830 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7845175961236048e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1827325974978048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28286888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1802952827583744 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13740521555832 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1813564689387520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20674611 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13838866861568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13760944159232 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1813564672348160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13761285590528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28286604 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13761285590528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95920180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20674614 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95920178 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚975 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394002391900160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117232810 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213163147 trainer/global_step‚11849 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117242967 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13740180994446 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48961502 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2ÏKEû24É&/ _wandbruntime‚10769  _runtime‚10769âê×óÊ&Š 0loss/FBt1/detection_loss‚0.41874691435694694 epoch‚29 3loss/SRqqt1/assignment_loss‚0.44074600249548795 3loss/FBt1/assignment_loss‚0.0009333474315037051 0loss/FRt1/detection_loss‚0.43034772816772754 _step‚976 0loss/FRt2/assignment_loss‚0.7356095152776598 /loss/FBt2/detection_loss‚0.5315097065122668 %loss/total_loss‚7.583108911959776 0loss/FRt1/assignment_loss‚1.6081764304963126 3loss/FBt2/assignment_loss‚0.0003929419423021569 /loss/FRt2/detection_loss‚1.5587227218503195 1loss/SRqqt1/detection_loss‚0.6809443067391648 1loss/SRqqt2/detection_loss‚0.6608153421095468 $ _timestamp‚1.784517596125443e+09 _runtime‚10769.700829094 2loss/SRqqt2/assignment_loss‚0.5161639545805374 trainer/global_step‚11849%o´Ë& 0loss/FRt2/assignment_loss‚0.7356095152776598 /loss/FBt2/detection_loss‚0.5315097065122668 %loss/total_loss‚7.583108911959776 0loss/FRt1/assignment_loss‚1.6081764304963126 3loss/FBt2/assignment_loss‚0.0003929419423021569 /loss/FRt2/detection_loss‚1.5587227218503195 1loss/SRqqt1/detection_loss‚0.6809443067391648 1loss/SRqqt2/detection_loss‚0.6608153421095468 $ _timestamp‚1.784517596125443e+09 _runtime‚10769.700829094 2loss/SRqqt2/assignment_loss‚0.5161639545805374 trainer/global_step‚11849 0loss/FBt1/detection_loss‚0.41874691435694694 epoch‚29 3loss/SRqqt1/assignment_loss‚0.44074600249548795 3loss/FBt1/assignment_loss‚0.0009333474315037051 0loss/FRt1/detection_loss‚0.43034772816772754 _step‚976Ðår»MpÌ&‚(:å ܧöÒÃÀúË$ memory_percent‚96.99828624725342& disk.nvme0n1.out‚67036.90966796875 disk.dm-0.in‚525.75( proc.memory.availableMB‚ 491.80078125 proc.cpu.threads‚17 network.recv‚ 2.8187e+06% disk./.usageGB‚1169.6158599853516" disk.nvme1c1n1.in‚ 229.87109375# disk.dm-0.out‚84784.67236328125" disk.dm-1.out‚13908.8017578125( disk.nvme0c0n1.out‚67036.90966796875) proc.memory.percent‚18.46795082092285 cpu‚25.68798979006036) disk./.usagePercent‚76.18406513501719 disk.nvme0n1.in‚ 295.87890625$ disk.nvme1c1n1.out‚ 32851.2734375 disk.nvme1n1.in‚ 229.87109375" disk.nvme1n1.out‚ 32851.2734375 disk.md0.in‚0" disk.nvme0c0n1.in‚ 295.87890625" proc.memory.rssMB‚ 3025.7890625 network.sent‚ 1.8881338e+07 disk.md0.out‚574.9375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0Ùňæ‘Í&‚(:† ܧöÒ®™ìÌ gpu.0.gpu‚44.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚290.61' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.87 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ÁåîÎ&‚(:’ ë§öÒÅ•µÌ gpu.0.gpu‚32.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.489' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.49633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-IS¨uÏ&‚(:ê ë§öÒ½»ÕÎ disk.rbd0.out‚0 disk.dm-1.in‚0" disk.dm-1.out‚13908.8876953125$ memory_percent‚96.99835777282715) proc.memory.percent‚18.46795082092285 network.recv‚ 2.823321e+06& disk.nvme0n1.out‚67202.39697265625" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-0.out‚ 84977.4296875" proc.memory.rssMB‚ 3025.7890625 cpu‚25.688154975490438( disk.nvme0c0n1.out‚67202.39697265625 disk.nvme0n1.in‚ 295.87890625& disk.nvme1n1.out‚32879.23779296875 disk.md0.out‚ 575.21875' proc.memory.availableMB‚ 491.7890625 proc.cpu.threads‚17 network.sent‚ 1.8931706e+07% disk./.usageGB‚1169.6144981384277" disk.nvme0c0n1.in‚ 295.87890625( disk.nvme1c1n1.out‚32879.23779296875 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚525.75' disk./.usagePercent‚76.183976429795 disk.rbd0.in‚0RäìuÐ&‚(:ê ú§öÒóûÌË& disk.nvme1c1n1.out‚32879.998046875 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-0.in‚525.75$ memory_percent‚96.99878692626953 network.sent‚ 1.8937013e+07 disk.rbd0.in‚0$ disk.nvme1n1.out‚32879.998046875 disk.dm-0.out‚84977.96484375% proc.memory.availableMB‚ 491.71875) proc.memory.percent‚18.46795082092285( disk./.usagePercent‚76.1839741935289" disk.nvme0c0n1.in‚ 295.87890625 disk.md0.out‚ 575.75390625# disk.dm-1.out‚13909.00732421875( disk.nvme0c0n1.out‚67203.42431640625 disk.nvme0n1.in‚ 295.87890625& disk.nvme0n1.out‚67203.42431640625" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-1.in‚0" proc.memory.rssMB‚ 3025.7890625 cpu‚25.688325348081644 proc.cpu.threads‚17 network.recv‚ 2.827111e+06% disk./.usageGB‚1169.6144638061523 disk.rbd0.out‚0Š¿m‘Ñ&‚(:† ú§öÒ餒Ì gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚240.81' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚80.27 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾X34Ò&/ _wandbruntime‚10813  _runtime‚10813ÑBËÓ&Å trainer/global_step‚11899 % _timestamp‚1.7845176401246734e+09 _runtime‚10813.700265019 _step‚977 ' lr-AdamW/pg1‚0.00022460470760707425 ' lr-AdamW/pg2‚0.00022460470760707425ÍY1BÐÔ&Ê _step‚977 ' lr-AdamW/pg1‚0.00022460470760707425 ' lr-AdamW/pg2‚0.00022460470760707425 trainer/global_step‚11899 % _timestamp‚1.7845176401246734e+09 _runtime‚10813.700265019Ñ„Ðs(4Õ&/ _wandbruntime‚10813  _runtime‚10813vä'(Ö&P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117707806 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1823971469070114 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20759577 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚11899 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1820830139992576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20759580 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1820830122953216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395628243746304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117717949 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1834645884578816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13815744586240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13893463556608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214009841 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96302037 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13794896380876 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1834645526115328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96302035 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1810176930289872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13794555819602 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1820830122953216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1834645526115328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409521688690688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1810176913250512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚978 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1820830139992576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96302037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13815403162112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1823971826670748 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49158533 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13893566728704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117717949 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845176401257343e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409521810475008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117707806 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13815403162112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1834645526115328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96302035 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1810176930289872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13794555819602 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1820830122953216 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1834645526115328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409521688690688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1810176913250512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚978 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1820830139992576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96302037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 XEDeviceStatsMonitor.on_train_batch_start/active_—,å ‡bytes.small_pool.freed‚13815403162112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1823971826670748 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49158533 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13893566728704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117717949 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845176401257343e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409521810475008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117707806 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13815403162112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117707806 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1823971469070114 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20759577 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚11899 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1820830139992576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20759580 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1820830122953216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395628243746304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117717949 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1834645884578816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13815744586240 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13893463556608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647Òÿq•l4Ø&/ _wandbruntime‚10814  _runtime‚10814‡ñ­'Ù&‹N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1824120008125838 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13796004769240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13816854692352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28400913 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117727619 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13816513261056 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1834795269187072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1820978397455360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20761314 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96309830 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13816513261056 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13816854692352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚395661405404160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1820978414494720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13795664207854 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214027290 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13894683930112 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1810324360957344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845176407515435e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117717462 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96309830 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1824120365726584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214027290 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49162224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214037449 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1820978414494720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _runtime‚10814.327342443 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96309828 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214037449 trainer/global_step‚11899 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117727619 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1810324343917984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49162515 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1834795269187072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚395661424016384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28401201 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚409556107946496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13894574473728 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1834795269187072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1820978397455360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20761314 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96309830 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13816513261056 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13816854692352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚395661405404160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1820978414494720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13795664207854 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214027290 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13894683930112 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1810324360957344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845176407515435e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117717462 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96309830 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1824120365726584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214027290 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49162224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214037449 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1820978414494720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _runtime‚10814.327342443 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96309828 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214037449 trainer/global_step‚11899 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117727619 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1810324343917984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49162515 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1834795269187072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚395661424016384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28401201 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚409556107946496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13894574473728 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1842114820324352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1842115178787840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20846277 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118192458 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845176846053312e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1817548446624112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1817548463663472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚411236656380928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118202601 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚411236528305152 trainer/global_step‚11949 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚982 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118192458 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96691685 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397287257250304 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1828243865099776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28513321 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118202601 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13871313688064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1842114820324352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49359306 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96691685 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13870972263936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96691687 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20846280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28513029 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13850379594284 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1828243865099776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1842115178787840 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1828243848060416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1831398485657122 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13949399130624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49359601 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1828243848060416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397287238638080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13949289667072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1831398843257756 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚10858.1811136735\% (é&†P YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118192458 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845176846053312e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1817548446624112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1817548463663472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚411236656380928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118202601 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚411236528305152 trainer/global_step‚11949 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚982 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118192458 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96691685 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397287257250304 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1828243865099776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28513321 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118202601 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13871313688064 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1842114820324352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49359306 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96691685 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13870972263936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96691687 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20846280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28513029 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13850379594284 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1828243865099776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1842115178787840 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1828243848060416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1831398485657122 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13949399130624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49359601 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1828243848060416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397287238638080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13949289667072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1831398843257756 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚10858.181113673 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96691687 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13850039033010 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13871313688064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1842114820324352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1842115178787840 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20846277 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224Öw…õ4ê&/ _wandbruntime‚10858  _runtime‚10858¾Æ™†'ë&‹N YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118202114 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96699480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20848014 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚397320437520384 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13851487982648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214911751 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49363607 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1828392139601920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118212271 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96699478 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49363311 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118212271 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1842264204925440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214901592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13872082362880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214911751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20848011 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1828392122562560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1842264563396096 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1828392122562560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28515593 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚411270822756864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1831547024712846 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118202114 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13872423794176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1831547382313592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845176852358756e+09 _step‚983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28515300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚411270952922624 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1817695877291584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96699478 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13872423794176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _runtime‚10858.811703671 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1842264563396096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96699480 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ðNa^» ì&N QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13872423794176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1831547382313592 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845176852358756e+09 _step‚983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28515300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚411270952922624 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1817695877291584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96699478 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13872423794176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E?DeviceStatsMonitor.on_train_batch_end/active.NÐ[large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _runtime‚10858.811703671 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1842264563396096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96699480 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118202114 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96699480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20848014 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚397320437520384 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13851487982648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214911751 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49363607 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1828392139601920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118212271 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96699478 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49363311 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118212271 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1842264204925440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214901592 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13872082362880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214911751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20848011 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1828392122562560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1842264563396096 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1828392122562560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28515593 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚411270822756864 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1831547024712846 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118202114 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13926882789888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398946270754304 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1835657590206976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1849584472996864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1849584114533376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14005115822080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118677110 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215768590 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28627902 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97081337 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13905862807692 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412951498137088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1838825859844764 % _timestamp‚1.7845177277112553e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118687253 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1849584472996864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1835657573167616 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118677110 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0£¹"û&†P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1849584114533376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118687253 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 trainer/global_step‚11999 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13905522246418 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97081337 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13926882789888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13926541365760 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49561180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215758445 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118677110 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215768590 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28627902 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97081337 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13905862807692 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412951498137088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1838825859844764 % _timestamp‚1.7845177277112553e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118687253 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1849584472996864 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1835657573167616 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118677110 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97081335 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412951367964160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14005227382784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚986 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13926882789888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚398946270754304 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1835657590206976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1849584472996864 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1849584114533376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14005115822080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97089128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚11999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118696923 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚398979451024384 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1838974398900600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118696923 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97089130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13927992896000 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1849733857605120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20934711 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14006232461312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _runtime‚10901.899253897 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1835805864709120 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845177283235345e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1835805847669760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49565167 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97089128 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1835805847669760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118686766 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13927651464704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215786053 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20934714 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚398979432412160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1825067410665184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215786053 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13927651464704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚987 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1849733499134464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1838974041299854 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13906630634670 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1825067427704544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49564875 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14006341917696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215775894 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97089130ò‰'þ&N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1835805864709120 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845177283235345e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1835805847669760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49565167 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97089128 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1835805847669760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118686766 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13927651464704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215786053 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20934714 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚398979432412160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1825067410665184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215786053 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13927651464704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚987 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1849733499134464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1838974041299854 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13906630634670 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1825067427704544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49564875 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14006341917696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215775894 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97089130 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412985664873472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1849733857605120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412985792942080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97089128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚11999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118696923 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚398979451024384 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1838974398900600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118696923 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97089130 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13927992896000 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1849733857605120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20934711 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14006232461312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _runtime‚10901.899253897Ûû54ÿ&/ _wandbruntime‚10901  _runtime‚10901€B5 €'… trainer/global_step‚11999 _runtime‚10901.899728118 /loss/FRt2/assignment_loss‚0.708896209963833 2loss/SRqqt1/assignment_loss‚0.4212104797281649 2loss/SRqqt2/assignment_loss‚0.5086094199638957 1loss/SRqqt2/detection_loss‚0.6439010449818967 %loss/total_loss‚7.639170593267286 % _timestamp‚1.7845177283244915e+09 _step‚988 /loss/FRt1/assignment_loss‚1.549399996176362 2loss/FBt1/assignment_loss‚0.004773671839705714 /loss/FRt1/detection_loss‚0.4873056630252436 .loss/FRt2/detection_loss‚1.603230510929218 1loss/SRqqt1/detection_loss‚0.6140852423914316 0loss/FBt1/detection_loss‚0.37320479143440255 3loss/FBt2/assignment_loss‚0.0002558161897077582 /loss/FBt2/detection_loss‚0.7242977466434241 epoch‚29ô·‡H'Š 2loss/SRqqt2/assignment_loss‚0.5086094199638957 1loss/SRqqt2/detection_loss‚0.6439010449818967 %loss/total_loss‚7.639170593267286 % _timestamp‚1.7845177283244915e+09 _step‚988 /loss/FRt1/assignment_loss‚1.549399996176362 2loss/FBt1/assignment_loss‚0.004773671839705714 /loss/FRt1/detection_loss‚0.4873056630252436 .loss/FRt2/detection_loss‚1.603230510929218 1loss/SRqqt1/detection_loss‚0.6140852423914316 0loss/FBt1/detection_loss‚0.37320479143440255 3loss/FBt2/assignment_loss‚0.0002558161897077582 /loss/FBt2/detection_loss‚0.7242977466434241 epoch‚29 trainer/global_step‚11999 _runtime‚10901.899728118 /loss/FRt2/assignment_loss‚0.708896209963833 2loss/SRqqt1/assignment_loss‚0.4212104797281649Üd\q“‚'‚(:ˆ ã¨öÒ´ÐÂË gpu.0.gpu‚72.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.747' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.249 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öQ£wƒ'‚(:ì ã¨öÒÛ¿ƒÍ disk.md0.out‚ 578.4609375 disk.dm-1.out‚ 13912.7578125" proc.memory.rssMB‚ 2918.6328125 proc.cpu.threads‚17$ disk./.usageGB‚1169.546085357666" disk.nvme0c0n1.in‚ 295.87890625 disk.nvme0n1.in‚ 295.87890625' disk.nvme1c1n1.out‚32927.6865234375 disk.nvme1n1.in‚ 229.87109375( proc.memory.availableMB‚ 488.26171875 network.sent‚ 1.9092369e+07' disk./.usagePercent‚76.179520296868& disk.nvme0n1.out‚67642.81982421875% disk.nvme1n1.out‚32927.6865234375# disk.dm-0.out‚85455.44580078125* proc.memory.percent‚17.813920974731445 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-0.in‚525.75 disk.dm-1.in‚0$ memory_percent‚97.01988697052002 cpu‚25.689712477597688 network.recv‚ 2.867171e+06( disk.nvme0c0n1.out‚67642.81982421875î×w4„'/ _wandbruntime‚10918  _runtime‚10918”Ø«´x …'õz =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8849741816520691 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7038216590881348 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9159258008003235 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08965208381414413 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5575000047683716 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.10002080351114273 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9182137250900269 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895260334014893 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7733511328697205 trainer/global_step‚11999 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23470349609851837 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6350623965263367 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8781178593635559 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_1‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9744720458984375 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.29267850518226624 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2844051718711853 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6828285455703735 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8518518209457397 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3363450765609741 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46549639105796814 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974562525749207 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7276015877723694 *jet/accuracy_1_of_1‚0.3532552719116211 .particle/accuracy_4_of_4‚0.952240526676178 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4427083432674408 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6985865831375122 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.435321182012558 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.33928701281547546 particle/accuracy_2_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6034553050994873 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7133679986000061 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.6505225300788879 particle/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.32939404249191284 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21324899792671204 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6606367826461792 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9956153631210327 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7683994770050049 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941510558128357 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6929042339324951 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6897339224815369 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07755431532859802 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 *jet/accuracy_1_of_3‚0.9248353242874146 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1985350400209427 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6007920503616333 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.15064489841461182 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19488829374313354 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6697838306427002 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6233906149864197 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3021465241909027 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24799321591854095 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9661802649497986 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31763067841529846 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9947911500930786 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6599019169807434 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_5‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5939522981643677 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.725447952747345 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2903345823287964 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 jet/accuracy_5_of_5‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22289146482944489 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44810667634010315 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6575009226799011 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5337494015693665 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7057292461395264 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964495897293091 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0889800637960434 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6877584457397461 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.056412480771541595 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8518518209457397 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36503511667251587 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5855554938316345 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6860709190368652 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3549671173095703 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8896069526672363 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6654691100120544 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3391845226287842 7 Purity/1FRt1SRqqt2FBt/FRtkP_î_purity‚0.8888888955116272 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6413226127624512 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27831798791885376 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.29422229528427124 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9952465295791626 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3286406099796295 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8564803004264832 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961268901824951 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6117788553237915 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960317015647888 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6018339395523071 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7158130407333374 6 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.089848592877388 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04479321092367172 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8784958124160767 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2576877772808075 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6252447962760925 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.680644154548645 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08429037779569626 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7733511328697205 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6510220170021057 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5865381956100464 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988343119621277 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982815980911255 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2918637692928314 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6411534547805786 *jet/accuracy_2_of_3‚0.5573519468307495 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6401247978210449 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6491238474845886 particle/accuracy_3_of_3‚1 _runtime‚10918.619932215 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3010687530040741 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6018339395523071 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.13255634903907776 jet/accuracy_1_of_5‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3291230797767639 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6149654984474182 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3639759123325348 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.48268693685531616 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3286406099796295 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.18500001728534698 *jet/accuracy_1_of_2‚0.7904742956161499 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7845177450439167e+09 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3812689483165741 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3201228678226471 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9466056823730469 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6151825785636902 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 _step‚989 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6905859112739563 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46788525581359863 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8777251243591309 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872360229492188 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7946144342422485 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22506098449230194 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7178378105163574 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3407328128814697 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6765655279159546 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_2_of_4‚0.7599518895149231 (loss/val_total_loss‚7.51310433273487 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9858534932136536 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09092244505882263 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06882674247026443 +particle/specificity‚0.9128105640411377 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9874468445777893 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3903392553329468 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 epoch‚29 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8744330406188965 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8721655011177063 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07804327458143234 +jet/accuracy_2_of_2‚0.16219574213027954 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35971641540527344 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6654216051101685 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08429037779569626 particle/accuracy_1_of_3‚1Èá4f€=†'úz ("Purity/2FRt1SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8564803004264832 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9961268901824951 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6117788553237915 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960317015647888 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6018339395523071 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7158130407333374 6 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.089848592877388 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04479321092367172 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8784958124160767 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2576877772808075 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6252447962760925 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.680644154548645 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08429037779569626 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7733511328697205 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6510220170021057 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5865381956100464 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988343119621277 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9982815980911255 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2918637692928314 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6411534547805786 *jet/accuracy_2_of_3‚0.5573519468307495 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6401247978210449 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6491238474845886 particle/accuracy_3_of_3‚1 _runtime‚10918.619932215 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3010687530040741 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6018339395523071 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.13255634903907776 jet/accuracy_1_of_5‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3291230797767639 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6149654984474182 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3639759123325348 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.48268693685531616 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3286406099796295 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.18500001728534698 *jet/accuracy_1_of_2‚0.7904742956161499 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7845177450439167e+09 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3812689483165741 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.3201228678226471 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9466056823730469 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6151825785636902 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 _step‚989 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6905859112739563 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46788525581359863 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8777251243591309 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872360229492188 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7946144342422485 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22506098449230194 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7178378105163574 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3407328128814697 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6765655279159546 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_2_of_4‚0.7599518895149231 (loss/val_total_loss‚7.51310433273487 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9858534932136536 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09092244505882263 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.06882674247026443 +particle/specificity‚0.9128105640411377 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9874468445777893 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3903392553329468 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 epoch‚29 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8744330406188965 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8721655011177063 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.07804327458143234 +jet/accuracy_2_of_2‚0.16219574213027954 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35971641540527344 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6654216051101685 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08429037779569626 particle/accuracy_1_of_3‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8849741816520691 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7038216590881348 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9159258008003235 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08965208381414413 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5575000047683716 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.10002080351114273 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9182137250900269 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9895260334014893 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7733511328697205 trainer/global_step‚11999 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23470349609851837 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6350623965263367 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8781178593635559 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_1‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9744720458984375 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.29267850518226624 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2844051718711853 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6828285455703735 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8518518209457397 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3363450765609741 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46549639105796814 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974562525749207 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7276015877723694 *jet/accuracy_1_of_1‚0.3532552719116211 .particle/accuracy_4_of_4‚0.952240526676178 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4427083432674408 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6985865831375122 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.435321182012558 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.33928701281547546 particle/accuracy_2_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6034553050994873 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7133679986000061 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.6505225300788879 particle/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.32939404249191284 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21324899792671204 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6606367826461792 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9956153631210327 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7683994770050049 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941510558128357 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6929042339324951 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6897339224815369 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07755431532859802 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 *jet/accuracy_1_of_3‚0.9248353242874146 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1985350400209427 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6007920503616333 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.15064489841461182 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19488829374313354 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6697838306427002 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6233906149864197 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.3021465241909027 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.24799321591854095 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9661802649497986 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31763067841529846 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9947911500930786 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6599019169807434 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_5‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5939522981643677 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.725447952747345 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2903345823287964 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 jet/accuracy_5_of_5‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22289146482944489 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44810667634010315 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6575009226799011 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5337494015693665 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7057292461395264 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964495897293091 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0889800637960434 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6877584457397461 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.056412480771541595 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8518518209457397 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36503511667251587 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5855554938316345 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6860709190368652 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3549671173095703 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8896069526672363 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6654691100120544 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3391845226287842 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6413226127624512 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.27831798791885376 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.29422229528427124 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9952465295791626 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3286406099796295Ý´&Åx‡'‚(:í ò¨öÒ„çïË disk.md0.in‚0 disk.md0.out‚ 578.8828125# disk.dm-0.out‚85476.10986328125( disk.nvme0c0n1.out‚67663.78662109375 proc.memory.rssMB‚ 2950.90625 cpu‚25.69375559132685 network.sent‚ 1.9165565e+07 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚32928.482421875 disk.dm-0.in‚525.75" disk.dm-1.out‚13912.9541015625 disk.rbd0.out‚0( proc.memory.availableMB‚ 455.97265625" disk.nvme0c0n1.in‚ 295.87890625 disk.nvme0n1.in‚ 295.87890625& disk.nvme0n1.out‚67663.78662109375& disk.nvme1c1n1.out‚32928.482421875* proc.memory.percent‚18.010902404785156 proc.cpu.threads‚17 network.recv‚ 2.870376e+06) disk./.usagePercent‚76.18306527559385 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.21696376800537% disk./.usageGB‚1169.6005096435547˜D´}œˆ'‚(:‘ ò¨öÒ†«ìË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.705' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.56833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»OÝÔw‰'‚(:ì ©öÒ˜ÎÂË network.sent‚ 1.9209694e+07" disk.nvme0c0n1.in‚ 295.87890625& disk.nvme1n1.out‚32930.76220703125 disk.md0.out‚ 579.453125$ memory_percent‚97.21693992614746' proc.memory.availableMB‚ 455.9765625 proc.memory.rssMB‚ 2950.90625* proc.memory.percent‚18.010902404785156 disk.rbd0.in‚0% disk.nvme0n1.out‚67709.4853515625 proc.cpu.threads‚17 network.recv‚ 2.8746e+06 disk.nvme0n1.in‚ 295.87890625" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-0.in‚525.75# disk.dm-0.out‚85522.70849609375 disk.dm-1.in‚0 cpu‚25.694222452203817) disk./.usagePercent‚76.19920614739813% disk./.usageGB‚1169.8483123779297 disk.rbd0.out‚0' disk.nvme0c0n1.out‚67709.4853515625( disk.nvme1c1n1.out‚32930.76220703125 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0" disk.dm-1.out‚13913.1181640625óÊcžŠ'‚(:“ ©öÒÏ­šÌ gpu.0.gpu‚92.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚270.133' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.04433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÙëv‹'‚(:ë ©öÒܳËË disk.nvme0n1.in‚ 295.87890625& disk.nvme1c1n1.out‚32996.701171875$ disk.nvme1n1.out‚32996.701171875# disk.dm-0.out‚85710.72021484375' proc.memory.availableMB‚ 455.9609375 proc.cpu.threads‚17 network.recv‚ 2.877956e+06) disk./.usagePercent‚76.19496096890711% disk./.usageGB‚1169.7831382751465 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 579.7109375 cpu‚25.694868336545714 network.sent‚ 1.9215633e+07% disk.nvme0n1.out‚67832.2412109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚525.75# memory_percent‚97.2170352935791 disk.dm-1.in‚0# disk.dm-1.out‚13913.22705078125' disk.nvme0c0n1.out‚67832.2412109375 disk.md0.in‚0 proc.memory.rssMB‚ 2950.90625* proc.memory.percent‚18.010902404785156 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 295.87890625NV4#œŒ'‚(:‘ ©öÒ¦¡Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚270.974' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.32466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚·[4'/ _wandbruntime‚10960  _runtime‚10960º†ŸËŽ'Å _runtime‚10960.858314186 _step‚990 ' lr-AdamW/pg1‚0.00021751447889147108 ' lr-AdamW/pg2‚0.00021751447889147108 trainer/global_step‚12049 % _timestamp‚1.7845177872827325e+09ÃÂLÐ'Ê trainer/global_step‚12049 % _timestamp‚1.7845177872827325e+09 _runtime‚10960.858314186 _step‚990 ' lr-AdamW/pg1‚0.00021751447889147108 ' lr-AdamW/pg2‚0.00021751447889147108Þù‹ý4'/ _wandbruntime‚10960  _runtime‚10960µèt€ä‘'P UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97593793 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119253399 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1834212769379536 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13981649869997 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13981309308723 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1845013642150400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚414709553984512 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1859016070453760 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1859016070453760 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1845013659189760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14081709676032 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1848194078688259 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1845013659189760 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1859016428917248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28778395 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14002428303360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14081814945280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚991 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1834212786418896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14002769727488 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.alloc2‡š#ated‚97593795 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1845013642150400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14002428303360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216847194 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21023793 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49802191 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚400627862920704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚400627844308480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216837049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97593795 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28778108 _runtime‚10960.862000202 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21023796 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚414709677865984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119243256 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚991 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1834212786418896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14002769727488 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97593795 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1845013642150400 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14002428303360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216847194 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21023793 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49802191 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚400627862920704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚400627844308480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216837049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97593795 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28778108 _runtime‚10960.862000202 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21023796 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚414709677865984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119243256 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚414709553984512 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1859016070453760 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1859016070453760 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1845013659189760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472ߥd4“'/ _wandbruntime‚10961  _runtime‚10961ä|o'”'ŠN JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97601586 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚414743861911552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49806185 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28780399 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚400661043190784 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119252912 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14082927915008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13982417696975 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216864657 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14003879833600 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13982758258361 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1859165813525504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21025527 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14082837332992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21025530 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97601586 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119263069 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49805926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216854498 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1845161916652544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14003879833600 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119252912 % _timestamp‚1.7845177878950431e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119263069 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97601588 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97601588 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1848342975344729 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚400661024578560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109194240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _step‚992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚414743971105792 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1845161933691904 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14003538402304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1845161916652544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1859165455054848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216864657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14003538402304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1845161916652544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1859165455054848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216864657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28780399 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚400661043190784 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119252912 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14082927915008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13982417696975 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216864657 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14003879833600 FDeviceStatsMonitor.on_trainE쎫 _batch_end/requested_bytes.all.peak‚ 23704151647 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13982758258361 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1859165813525504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21025527 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14082837332992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21025530 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97601586 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119263069 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49805926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216854498 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1845161916652544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14003879833600 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119252912 % _timestamp‚1.7845177878950431e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119263069 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97601588 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97601588 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528àX”CÄ4–'/ _wandbruntime‚10961  _runtime‚10961Sbv—'ˆ 0loss/FRt2/assignment_loss‚0.7183644161948675 %loss/total_loss‚7.487345147834612 _runtime‚10961.472570827 _step‚993 0loss/FRt1/assignment_loss‚1.6020830427199926 0loss/FRt1/detection_loss‚0.47817915274022615 /loss/FRt2/detection_loss‚1.5818266643741745 1loss/SRqqt1/detection_loss‚0.7230395063452895 /loss/FBt1/detection_loss‚0.5197360633913103 % _timestamp‚1.7845177878971124e+09 3loss/SRqqt1/assignment_loss‚0.40532218803605796 1loss/SRqqt2/assignment_loss‚0.478508618154648 3loss/FBt1/assignment_loss‚0.0004001907067039577 2loss/FBt2/assignment_loss‚0.045990759304120526 /loss/FBt2/detection_loss‚0.2757390112679262 epoch‚30 1loss/SRqqt2/detection_loss‚0.6581555345992944 trainer/global_step‚12049™áѹ˜' 3loss/SRqqt1/assignment_loss‚0.40532218803605796 1loss/SRqqt2/assignment_loss‚0.478508618154648 3loss/FBt1/assignment_loss‚0.0004001907067039577 2loss/FBt2/assignment_loss‚0.045990759304120526 /loss/FBt2/detection_loss‚0.2757390112679262 epoch‚30 1loss/SRqqt2/detection_loss‚0.6581555345992944 trainer/global_step‚12049 0loss/FRt2/assignment_loss‚0.7183644161948675 %loss/total_loss‚7.487345147834612 _runtime‚10961.472570827 _step‚993 0loss/FRt1/assignment_loss‚1.6020830427199926 0loss/FRt1/detection_loss‚0.47817915274022615 /loss/FRt2/detection_loss‚1.5818266643741745 1loss/SRqqt1/detection_loss‚0.7230395063452895 /loss/FBt1/detection_loss‚0.5197360633913103 % _timestamp‚1.7845177878971124e+09ááö’q™'‚(:æ Ÿ©öÒüóË network.recv‚ 2.8814e+06 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚32997.6787109375 disk.nvme0n1.in‚ 295.87890625" disk.nvme1c1n1.in‚ 229.87109375( proc.memory.availableMB‚ 455.89453125* proc.memory.percent‚18.010902404785156% disk./.usageGB‚1169.7839126586914 disk.dm-0.in‚525.75 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 295.87890625 proc.memory.rssMB‚ 2950.90625) disk./.usagePercent‚76.19501140913151 disk.md0.in‚0 disk.md0.out‚ 580.03125# disk.dm-1.out‚13913.28955078125( disk.nvme0c0n1.out‚67837.21728515625& disk.nvme0n1.out‚67837.21728515625$ memory_percent‚97.21744060516357 proc.cpu.threads‚17 network.sent‚ 1.9233353e+07 disk.dm-0.out‚ 85715.90625 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme1c1n1.out‚32997.6787109375 cpu‚25.69539740644363“¦$žš'‚(:“ Ÿ©öÒÇó»Ì gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚263.243' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.74766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡9\íw›'‚(:ì ®©öÒÖ‡ Ë disk.rbd0.out‚0 cpu‚25.695225026655386 network.recv‚ 2.885225e+06& disk.nvme1n1.out‚33001.72021484375 disk.md0.in‚0 disk.md0.out‚ 580.421875 disk.dm-0.out‚85717.43359375 disk.rbd0.in‚0* proc.memory.percent‚18.010902404785156% disk./.usageGB‚1169.7837409973145" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚525.75" disk.nvme0c0n1.in‚ 295.87890625( disk.nvme0c0n1.out‚67838.93310546875( proc.memory.availableMB‚ 455.87109375 proc.memory.rssMB‚ 2950.90625 disk.nvme0n1.in‚ 295.87890625& disk.nvme0n1.out‚67838.93310546875 disk.dm-1.in‚0! disk.dm-1.out‚13916.673828125$ memory_percent‚97.21758365631104 proc.cpu.threads‚17 network.sent‚ 1.928185e+07) disk./.usagePercent‚76.19500022780097( disk.nvme1c1n1.out‚33001.72021484375qœïxœœ'‚(:‘ ®©öÒ³Ò÷Ë gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.049' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.34966666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø¿,Ø{'‚(:𠽩öÒª‡°Ë disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚33003.65576171875 network.recv‚ 2.888935e+06 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 295.87890625( disk.nvme0c0n1.out‚67845.49072265625 disk.md0.out‚ 580.98046875* proc.memory.percent‚18.010902404785156 cpu‚25.69489035276574 proc.cpu.threads‚17) disk./.usagePercent‚76.19499997932697( disk.nvme1c1n1.out‚33003.65576171875 disk.md0.in‚0" disk.dm-0.out‚85724.6728515625 disk.dm-1.in‚0! disk.dm-1.out‚13916.744140625$ memory_percent‚97.21758365631104 proc.memory.rssMB‚ 2950.90625 network.sent‚ 1.9287928e+07% disk./.usageGB‚1169.7837371826172 disk.nvme0n1.in‚ 295.87890625& disk.nvme0n1.out‚67845.49072265625 disk.dm-0.in‚525.75( proc.memory.availableMB‚ 455.87109375`Úž'‚(:’ ½©öÒæÀ‰Ì gpu.0.gpu‚10.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.052' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.35066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~éÜj4Ÿ'/ _wandbruntime‚11003  _runtime‚11003m5?Ê 'Ä _step‚994 ' lr-AdamW/pg1‚0.00021516239368945557 ' lr-AdamW/pg2‚0.00021516239368945557 trainer/global_step‚12099 $ _timestamp‚1.784517829873253e+09 _runtime‚11003.450072465vŽZYÏ¡'É trainer/global_step‚12099 $ _timestamp‚1.784517829873253e+09 _runtime‚11003.450072465 _step‚994 ' lr-AdamW/pg1‚0.00021516239368945557 ' lr-AdamW/pg2‚0.00021516239368945557âóY¥ç4¢'/ _wandbruntime‚11003  _runtime‚11003Õ»ÀK(£'€P VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109201408 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1866485364662784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14058338829312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90589184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97983445 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97983443 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119738051 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119738051 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1866485364662784 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1852427367257600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97983445 trainer/global_step‚12099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119727908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚416424557245440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚402286857812480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14036792522131 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217721496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50004544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21110493 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14057997405184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚402286876424704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14058338829312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1852427367257600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1852427384296960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21110496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14057997405184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28894048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1866485723126272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚995 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1852427384296960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50004282 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1855621095275267 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14037133083405 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1841584302753136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97983443 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145ò§: (¤'…P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97983443 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119738051 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119738051 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1866485364662784 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1852427367257600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97983445 trainer/global_step‚12099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119727908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚416424557245440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚402286857812480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14036792522131 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217721496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50004544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21110493 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14057997405184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚402286876424704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14058338829312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1852427367257600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1852427384296960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21110496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14057997405184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28894048 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1866485723126272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚995 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1852427384296960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50004282 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1855621095275267 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14037133083405 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1841584302753136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97983443 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109201408 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1866485364662784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14058338829312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90589184 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97983445 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184ã2š 4¥'/ _wandbruntime‚11004  _runtime‚11004Ã8M ¦'‰N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14059448935424 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1866635107734528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1841731750459968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1852575658799104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14059448935424 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28896335 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14138699952640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119747721 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1852575641759744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14038241471769 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚996 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1852575641759744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50008282 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14059107504128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50008565 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21112227 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocateÆ›01 d‚14138796826112 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1855769634330991 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97991236 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119737564 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14059107504128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217738959 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217728800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14037900910383 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1866635107734528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97991238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119737564 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1852575658799104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 trainer/global_step‚12099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217728800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚402320056694784 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1841731733420608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 $ _timestamp‚1.784517830485105e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚402320038082560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21112230 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚416458853520896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217738959 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217728800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14037900910383 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1866635107734528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97991238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119737564 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1852575658799104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 trainer/global_step‚12099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217728800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚402320056694784 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1841731733420608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 $ _timestamp‚1.784517830485105e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚402320038082560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21112230 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚416458853520896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28896335 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14138699952640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119747721 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1852575641759744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14038241471769 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚996 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1852575641759744 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50008282 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14059107504128ä²(h¸4¨'/ _wandbruntime‚11004  _runtime‚11004©7e± ©'„ epoch‚30 %loss/total_loss‚7.614883615688214 3loss/SRqqt1/assignment_loss‚0.40161931842008286 % _timestamp‚1.7845178304875114e+09 _runtime‚11004.062789436 _step‚997 0loss/FRt2/assignment_loss‚0.7365810038856647 1loss/SRqqt2/assignment_loss‚0.523610816695408 2loss/FBt1/assignment_loss‚0.000585629758313407 /loss/FRt2/detection_loss‚1.4434838514573323 1loss/SRqqt1/detection_loss‚0.5854624209203072 1loss/SRqqt2/detection_loss‚0.6448365714229258 /loss/FBt2/detection_loss‚0.5484546552149531 trainer/global_step‚12099 /loss/FRt1/assignment_loss‚1.526048525467815 0loss/FBt2/assignment_loss‚0.2182353432243719 0loss/FRt1/detection_loss‚0.44233670778306944 /loss/FBt1/detection_loss‚0.5436287714379702ËV#eª'‰ 0loss/FRt2/assignment_loss‚0.7365810038856647 1loss/SRqqt2/assignment_loss‚0.523610816695408 2loss/FBt1/assignment_loss‚0.000585629758313407 /loss/FRt2/detection_loss‚1.4434838514573323 1loss/SRqqt1/detection_loss‚0.5854624209203072 1loss/SRqqt2/detection_loss‚0.6448365714229258 /loss/FBt2/detection_loss‚0.5484546552149531 trainer/global_step‚12099 /loss/FRt1/assignment_loss‚1.526048525467815 0loss/FBt2/assignment_loss‚0.2182353432243719 0loss/FRt1/detection_loss‚0.44233670778306944 /loss/FBt1/detection_loss‚0.5436287714379702 epoch‚30 %loss/total_loss‚7.614883615688214 3loss/SRqqt1/assignment_loss‚0.40161931842008286 % _timestamp‚1.7845178304875114e+09 _runtime‚11004.062789436 _step‚997å÷já‹w«'‚(:ì Ì©öÒ´ž¹Ë disk.rbd0.out‚0 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚33004.63037109375# disk.dm-0.out‚85726.10498046875" disk.dm-1.out‚13916.9521484375 network.recv‚ 2.892222e+06 disk.md0.out‚ 581.3515625( proc.memory.availableMB‚ 455.83984375* proc.memory.percent‚18.010902404785156 proc.cpu.threads‚17$ disk./.usageGB‚1169.784267425537" disk.nvme0c0n1.in‚ 295.87890625( disk.nvme1c1n1.out‚33004.63037109375 disk.md0.in‚0 disk.dm-0.in‚525.75 cpu‚25.694624874549945 network.sent‚ 1.9292808e+07& disk.nvme0c0n1.out‚67846.955078125 disk.nvme0n1.in‚ 295.87890625$ disk.nvme0n1.out‚67846.955078125" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-1.in‚0$ memory_percent‚97.21777439117432 proc.memory.rssMB‚ 2950.90625( disk./.usagePercent‚76.1950345172146 disk.rbd0.in‚0 $“¬'‚(:ˆ Ì©öÒÉ•ÎÌ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚283.659' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.553 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïWÃÚu­'‚(:ê Û©öÒ±³ÎË disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 455.4765625 proc.memory.rssMB‚ 2950.90625 cpu‚25.69435966262518 network.recv‚ 2.897087e+06% disk./.usageGB‚1169.7842864990234 disk.md0.out‚ 581.58203125" disk.nvme0c0n1.in‚ 295.87890625" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚33005.27001953125 proc.cpu.threads‚17 network.sent‚ 1.935604e+07 disk.dm-0.in‚525.75% disk.nvme0c0n1.out‚67849.16796875# disk.nvme0n1.out‚67849.16796875 disk.nvme1n1.in‚ 229.87109375* proc.memory.percent‚18.010902404785156) disk./.usagePercent‚76.19503575958466# disk.dm-0.out‚85728.38916015625" disk.dm-1.out‚13916.9990234375 disk.nvme0n1.in‚ 295.87890625& disk.nvme1n1.out‚33005.27001953125$ memory_percent‚97.21999168395996ÚnÎä®'‚(:’ Û©öÒªì‘Ì gpu.0.gpu‚16.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.809' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.60300000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶\òSv¯'‚(:ë ê©öÒŸÓÈË" disk.nvme0c0n1.in‚ 295.87890625% disk.nvme0n1.out‚67849.9990234375 disk.nvme1n1.in‚ 229.87109375# disk.dm-0.out‚85729.02197265625 disk.dm-1.in‚0 proc.memory.rssMB‚ 2950.90625 cpu‚25.69402835854604 network.sent‚ 1.9360915e+07) disk./.usagePercent‚76.19502979620837 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚67849.9990234375 disk.nvme0n1.in‚ 295.87890625" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚33005.8271484375 network.recv‚ 2.90039e+06% disk.nvme1n1.out‚33005.8271484375 disk.md0.in‚0 disk.md0.out‚ 581.8671875# disk.dm-1.out‚13917.13134765625$ memory_percent‚97.21999168395996 proc.cpu.threads‚17$ disk./.usageGB‚1169.784194946289 disk.dm-0.in‚525.75' proc.memory.availableMB‚ 455.4765625* proc.memory.percent‚18.010902404785156ŒYÍ“°'‚(:ˆ ê©öÒ¸ŠÌ gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.617' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.539 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¼ðP4±'/ _wandbruntime‚11045  _runtime‚11045»ÍU˲'Å _step‚998 ' lr-AdamW/pg1‚0.00021281622276260714 ' lr-AdamW/pg2‚0.00021281622276260714 trainer/global_step‚12149 % _timestamp‚1.7845178719945405e+09 _runtime‚11045.570083752è¦WRг'Ê % _timestamp‚1.7845178719945405e+09 _runtime‚11045.570083752 _step‚998 ' lr-AdamW/pg1‚0.00021281622276260714 ' lr-AdamW/pg2‚0.00021281622276260714 trainer/global_step‚12149æV×4´'/ _wandbruntime‚11045  _runtime‚11045 .kú&µ'P XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1859841109404160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21197196 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1859841092364800 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98373095 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98373095 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50205444 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1873955017335296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29007940 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1848955853166096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14113566507008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14113907931136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120222703 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚999 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98373093 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14113566507008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418139303831040 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14092616296813 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1863048111862275 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚12149 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21197193 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120222703 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1873954658871808 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14092275735539 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14193544075264 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1863048469462909 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120212560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29008248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14113907931136 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.tþ½w current‚1180 _runtime‚11045.574046124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218595798 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98373093 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1_áZ (¶'†P PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98373095 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98373095 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50205444 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1873955017335296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29007940 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1848955853166096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14113566507008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14113907931136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120222703 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚999 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98373093 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14113566507008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418139303831040 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14092616296813 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1863048111862275 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚12149 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21197193 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120222703 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1873954658871808 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14092275735539 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14193544075264 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1863048469462909 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120212560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29008248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14113907931136 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚11045.574046124 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218595798 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98373093 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1859841109404160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21197196 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1859841092364800çûLòc4·'/ _wandbruntime‚11046  _runtime‚11046áî`'¸'ŒN OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120222216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚11046.478047727 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14114676605952 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98380886 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418173728428032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚307 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134360064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14093384123791 % _timestamp‚1.7845178729023066e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29010209 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1863196650917999 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403979051586560 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1859989366866944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14115018037248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98380886 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1859989383906304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21198927 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403979070198784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1874104401943552 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120222216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21198930 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14194658229248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218613261 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚1000 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1874104401943552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50209136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29010513 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1849103283833568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14093724685177 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218603102 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1863197008518745 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1859989383906304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50209443 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120232373 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218603102 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218613261 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1874104043472896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14115018037248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚307 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 134360064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14093384123791 % _timestamp‚1.7845178729023066e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29010209 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1863196650917999 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403979051586560 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1859989366866944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14115018037248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98380886 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1859989383906304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21198927 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403979070198784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1874104401943552 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120222216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21198930 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14194658229248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218613261 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚1000 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1874104401943552 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50209136 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 115747840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29010513 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1849103283833568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14093724685177 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218603102 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1863197008518745 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1859989383906304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50209443 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120232373 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218603102 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218613261 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1874104043472896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14115018037248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98380886 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418173728428032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14169135608832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120707355 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1881423953080832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845179133864703e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98762743 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14249250858496 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1856327386539696 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50406876 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚405604903432704 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29123273 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21283896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1867254834511360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 219470100 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29122983 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚12199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14169477032960 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1867254834511360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14169135608832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21283893 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98762745 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419854135678976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14147758948947 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98762745 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14249356127744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1870475486049917 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14169477032960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120697212 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1881423953080832 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120697212 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 219459955 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14169135608832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120707355 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1881423953080832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845179133864703e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98762743 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14249250858496 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1856327386539696 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50406876 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚405604903432704 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29123273 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21283896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1867254834511360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 219470100 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29122983 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚12199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14169477032960 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1867254834511360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14169135608832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21283893 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98762745 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419854135678976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14147758948947 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98762745 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14249356127744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1870475486049917 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14169477032960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120697212 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1881423953080832 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120697212ëj ã4É'/ _wandbruntime‚11087  _runtime‚11087ÅPk Ê'ŒN YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14170245707776 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1856474800167808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚1004 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1881573696152576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50411183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219487563 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21285627 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14250364036608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14149207898585 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚12199 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚419888429127168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14170587139072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 219477404 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98770538 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1867403109013504 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120706868 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21285630 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1867403109013504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1870624025105753 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14148867337199 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98770536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1881573337681920 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845179139900212e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50410887 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _runtime‚11087.565788366 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98770536 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219487563 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚405638065090560 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1856474817207168ŸºòK'Ë'‘N _runtime‚11087.565788366 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98770536 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219487563 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚405638065090560 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1856474817207168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14170245707776 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1856474800167808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚1004 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1881573696152576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50411183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219487563 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21285627 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14250364036608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14149207898585 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚12199 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚419888429127168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14170587139072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 219477404 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98770538 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1867403109013504 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120706868 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21285630 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1867403109013504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1870624025105753 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14148867337199 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98770536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1881573337681920 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845179139900212e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50410887 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452ì7‘ÄÉ4Ì'/ _wandbruntime‚11087  _runtime‚11087¡O¥Í'Š %loss/total_loss‚7.429507448155093 _runtime‚11087.575636669 2loss/SRqqt2/assignment_loss‚0.5077483923091405 2loss/FBt2/assignment_loss‚0.002824766904001187 /loss/FRt1/detection_loss‚0.4571285679893101 /loss/FRt2/detection_loss‚1.5050211724131104 1loss/SRqqt1/detection_loss‚0.6562184262359024 1loss/SRqqt2/detection_loss‚0.6172945678362571 3loss/SRqqt1/assignment_loss‚0.43849728122516496 2loss/FBt1/assignment_loss‚0.013056272167426641 epoch‚30 trainer/global_step‚12199 0loss/FRt1/assignment_loss‚1.6322336070845571 0loss/FRt2/assignment_loss‚0.7371041239151415 % _timestamp‚1.7845179139978926e+09 _step‚1005 0loss/FBt1/detection_loss‚0.41666072882643285 0loss/FBt2/detection_loss‚0.44571954124864743[e QÎ' 0loss/FRt1/assignment_loss‚1.6322336070845571 0loss/FRt2/assignment_loss‚0.7371041239151415 % _timestamp‚1.7845179139978926e+09 _step‚1005 0loss/FBt1/detection_loss‚0.41666072882643285 0loss/FBt2/detection_loss‚0.44571954124864743 %loss/total_loss‚7.429507448155093 _runtime‚11087.575636669 2loss/SRqqt2/assignment_loss‚0.5077483923091405 2loss/FBt2/assignment_loss‚0.002824766904001187 /loss/FRt1/detection_loss‚0.4571285679893101 /loss/FRt2/detection_loss‚1.5050211724131104 1loss/SRqqt1/detection_loss‚0.6562184262359024 1loss/SRqqt2/detection_loss‚0.6172945678362571 3loss/SRqqt1/assignment_loss‚0.43849728122516496 2loss/FBt1/assignment_loss‚0.013056272167426641 epoch‚30 trainer/global_step‚12199íâ‚ô|Ï'‚(:ñ ¦ªöÒ˜³ Ë network.recv‚ 2.914761e+06 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚33008.67138671875 disk.md0.out‚ 583.6015625 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2950.90625) disk./.usagePercent‚76.16409850885378 disk.md0.in‚0" disk.nvme0c0n1.in‚ 295.87890625( proc.memory.availableMB‚ 455.09765625 cpu‚25.692870156643924% disk./.usageGB‚1169.3093223571777 disk.dm-0.in‚525.75# disk.dm-0.out‚85738.41650390625 disk.dm-1.in‚0( disk.nvme0c0n1.out‚67860.68212890625& disk.nvme0n1.out‚67860.68212890625* proc.memory.percent‚18.010902404785156# disk.dm-1.out‚13917.56201171875 disk.nvme0n1.in‚ 295.87890625" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚33008.67138671875$ memory_percent‚97.22230434417725 proc.cpu.threads‚17 network.sent‚ 1.94373e+07úÖÌZžÐ'‚(:“ ¦ªöÒØψÌ gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚269.425' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.80833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\SsÿxÑ'‚(:í µªöÒá˼Ë disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 295.87890625% disk.nvme0n1.out‚67961.0654296875% disk.nvme1n1.out‚33011.7373046875$ memory_percent‚97.22235202789307% disk./.usageGB‚1169.9129219055176 disk.dm-1.in‚0 disk.nvme0n1.in‚ 295.87890625" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚33011.7373046875 proc.memory.rssMB‚ 2950.90625* proc.memory.percent‚18.010902404785156 cpu‚25.69258673450642 network.recv‚ 2.918813e+06 disk.md0.in‚0 disk.md0.out‚ 584.328125# disk.dm-0.out‚85840.18701171875 disk.rbd0.out‚0( proc.memory.availableMB‚ 455.08984375 proc.cpu.threads‚17# disk.dm-1.out‚13917.71826171875' disk.nvme0c0n1.out‚67961.0654296875 disk.nvme1n1.in‚ 229.87109375 network.sent‚ 1.9498845e+07) disk./.usagePercent‚76.20341455173539 disk.dm-0.in‚525.75í:.žÒ'‚(:“ µªöÒ߈‘Ì gpu.0.gpu‚40.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚255.226' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.07533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uecã4Ó'/ _wandbruntime‚11129  _runtime‚11129Ä´œÊÔ'Ä ' lr-AdamW/pg1‚0.00020814222610556797 ' lr-AdamW/pg2‚0.00020814222610556797 trainer/global_step‚12249 % _timestamp‚1.7845179561570287e+09 _runtime‚ 11129.7342287 _step‚1006À‡ ƒÏÕ'É ' lr-AdamW/pg1‚0.00020814222610556797 ' lr-AdamW/pg2‚0.00020814222610556797 trainer/global_step‚12249 % _timestamp‚1.7845179561570287e+09 _runtime‚ 11129.7342287 _step‚1006î(TD·4Ö'/ _wandbruntime‚11129  _runtime‚11129¼!úM(×'P OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421569077949440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1888893247289856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121181864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121181864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845179561594422e+09 _runtime‚11129.734935681 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121192007 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14224704710656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14225046134784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21370593 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50608288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _step‚1007 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1863698919913296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407263898324480 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14305161012736 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1863698902873936 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14225046134784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50608575 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99152395 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14224704710656 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1888893605753344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21370596 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421568962456576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1874668542579200 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1877902145036291 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1874668559618560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220344402 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121192007 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99152393 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚12249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1877902502636925 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14203242162355 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407263916936704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1888893247289856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14305064132096 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99152393 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29237695 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14203582723629 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220334257 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1888893247289856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121181864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121181864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845179561594422e+09 _runtime‚11129.734935681 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121192007 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14224704710656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14225046134784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21370593 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50608288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _step‚1007 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1863698919913296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407263898324480 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14305161012736 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1863698902873936 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14225046134784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50608575 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99152395 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14224704710656 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1888893605753344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21370596 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421568962456576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1874668542579200 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1877902145036291 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1874668559618560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220344402 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloÕ› Èc‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121192007 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99152393 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚12249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1877902502636925 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14203242162355 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407263916936704 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1888893247289856 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14305064132096 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99152393 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29237695 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14203582723629 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220334257 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121191520 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14204691111993 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99160188 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14226156240896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14306177644032 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220361865 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21372327 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14306282906112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1874816834120704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220351706 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1874816834120704 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14225814809600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50612342 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421603380112896 trainer/global_step‚12249 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220351706 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99160188 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50612636 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21372330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29240015 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14204350550607 _step‚1008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1874816817081344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14225814809600 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99160186 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407297078594560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121201677 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1863846350580768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1878051041692761 _runtime‚11130.365334912 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121191520 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14204691111993 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99160188 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14226156240896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14306177644032 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220361865 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21372327 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14306282906112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1874816834120704 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220351706 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1874816834120704 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14225814809600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50612342 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421603380112896 trainer/global_step‚12249 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220351706 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99160188 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50612636 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21372330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29240015 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14204350550607 _step‚1008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1874816817081344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14225814809600 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99160186 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407297078594560 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121201677 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1863846350580768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1878051041692761ð8°Á4Þ'/ _wandbruntime‚11130  _runtime‚11130oÅ© ß'† 0loss/FRt1/assignment_loss‚1.5241218601942297 /loss/FBt2/detection_loss‚0.5106873674783856 trainer/global_step‚12249 0loss/FRt2/assignment_loss‚0.7649286363013955 .loss/FRt1/detection_loss‚0.489027569699285 0loss/SRqqt1/detection_loss‚0.674321487655986 /loss/FBt1/detection_loss‚0.4190172225815984 %loss/total_loss‚7.409218163150939 _runtime‚11130.365991614 _step‚1009 2loss/SRqqt1/assignment_loss‚0.4723989927734432 2loss/FBt1/assignment_loss‚0.013470860448010468 1loss/SRqqt2/detection_loss‚0.5635820468810077 epoch‚30 3loss/SRqqt2/assignment_loss‚0.47215612964743436 3loss/FBt2/assignment_loss‚0.0007242384744199626 /loss/FRt2/detection_loss‚1.5047817510157433 $ _timestamp‚1.784517956790663e+09¾»à'‹ _step‚1009 2loss/SRqqt1/assignment_loss‚0.4723989927734432 2loss/FBt1/assignment_loss‚0.013470860448010468 1loss/SRqqt2/detection_loss‚0.5635820468810077 epoch‚30 3loss/SRqqt2/assignment_loss‚0.47215612964743436 3loss/FBt2/assignment_loss‚0.0007242384744199626 /loss/FRt2/detection_loss‚1.5047817510157433 $ _timestamp‚1.784517956790663e+09 0loss/FRt1/assignment_loss‚1.5241218601942297 /loss/FBt2/detection_loss‚0.5106873674783856 trainer/global_step‚12249 0loss/FRt2/assignment_loss‚0.7649286363013955 .loss/FRt1/detection_loss‚0.489027569699285 0loss/SRqqt1/detection_loss‚0.674321487655986 /loss/FBt1/detection_loss‚0.4190172225815984 %loss/total_loss‚7.409218163150939 _runtime‚11130.365991614ñúýrÖsá'‚(:è ÓªöÒôæ–Ë( proc.memory.availableMB‚ 455.08203125 network.recv‚ 2.926526e+06) disk./.usagePercent‚76.19504644396716% disk./.usageGB‚1169.7844505310059 disk.nvme1n1.in‚ 229.87109375" disk.nvme1n1.out‚ 33612.2109375# disk.dm-0.out‚86251.40185546875 cpu‚25.692876878372697 proc.cpu.threads‚17 disk.nvme0n1.in‚ 295.87890625$ disk.nvme0n1.out‚68308.236328125" disk.nvme1c1n1.in‚ 229.87109375$ disk.nvme1c1n1.out‚ 33612.2109375 disk.md0.in‚0 disk.md0.out‚ 584.97265625* proc.memory.percent‚18.010902404785156 disk.dm-0.in‚525.75 disk.rbd0.out‚0 proc.memory.rssMB‚ 2950.90625 network.sent‚ 1.9555467e+07 disk.dm-1.in‚0# disk.dm-1.out‚14452.73583984375 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 295.87890625& disk.nvme0c0n1.out‚68308.236328125$ memory_percent‚97.22239971160889™¸$‘â'‚(:† ÓªöÒðÉ‹Ì gpu.0.gpu‚96.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚121.41' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.47 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tÅØ{ã'‚(:ð âªöÒñÌâË disk.dm-1.in‚0 network.sent‚ 1.9609491e+07( disk.nvme1c1n1.out‚33612.87939453125 disk.md0.out‚ 585.25390625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 295.87890625 proc.memory.rssMB‚ 2950.90625 proc.cpu.threads‚17 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚525.75' disk.nvme0c0n1.out‚68309.5927734375 cpu‚25.692995428303824# disk.dm-0.out‚86252.66748046875 network.recv‚ 2.93063e+06) disk./.usagePercent‚76.01810810018918" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚33612.87939453125 disk.md0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚18.010902404785156% disk./.usageGB‚1167.0680046081543" disk.dm-1.out‚14452.8837890625" disk.nvme0c0n1.in‚ 295.87890625% disk.nvme0n1.out‚68309.5927734375$ memory_percent‚97.22092151641846( proc.memory.availableMB‚ 455.32421875à«Ažä'‚(:“ âªöÒ®½ªÌ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.301' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.10033333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºµôì4å'/ _wandbruntime‚11174  _runtime‚11174Éž4wÉæ'à trainer/global_step‚12299 $ _timestamp‚1.784518001084085e+09 _runtime‚11174.659772538 _step‚1010 & lr-AdamW/pg1‚0.0002058147005774482 & lr-AdamW/pg2‚0.00020581470057744827FÐGÎç'È _step‚1010 & lr-AdamW/pg1‚0.0002058147005774482 & lr-AdamW/pg2‚0.0002058147005774482 trainer/global_step‚12299 $ _timestamp‚1.784518001084085e+09 _runtime‚11174.659772538ò:‘_4è'/ _wandbruntime‚11174  _runtime‚11174–õŽé'‚P UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121676659 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29352852 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚423283832873472 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14258725375763 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99542045 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14280273812480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 % _timestamp‚1.7845180010849423e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1882082267686400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.cuèðØzrrent‚ 357600634 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1871070436247536 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14360921044992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121666516 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14259065937037 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99542043 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1011 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1882082284725760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1885329519223933 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21457296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121676659 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1896362899962368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚408922911828480 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14361024217088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 _runtime‚11174.660496698 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1882082284725760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21457293 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1896362899962368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 221218704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚408922930440704 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1896362541498880 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1885329161623299 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14280615236608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50810148 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1896362541498880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99542045 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121666516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50809869 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14280615236608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1871070453286896 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0è (ê'‡P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚423283832873472 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14258725375763 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99542045 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14280273812480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 % _timestamp‚1.7845180010849423e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1882082267686400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1871070436247536 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14360921044992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121666516 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14259065937037 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99542043 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1011 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1882082284725760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1885329519223933 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21457296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121676659 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1896362899962368 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚408922911828480 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14361024217088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 _runtime‚11174.660496698 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1882082284725760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21457293 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1896362899962368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 221218704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚408922930440704 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1896362541498880 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1885329161623299 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14280615236608 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50810148 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1896362541498880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99542045 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121666516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50809869 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14280615236608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1871070453286896 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121676659 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29352852 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528ó¯ökyvë'‚(:ë ñªöÒà®óË disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 585.84375* proc.memory.percent‚18.010902404785156 proc.cpu.threads‚17 network.recv‚ 2.93414e+06! disk.dm-0.out‚86278.123046875 disk.dm-1.in‚0" disk.dm-1.out‚14452.9462890625 disk.rbd0.in‚0 proc.memory.rssMB‚ 2950.90625 cpu‚25.69316054526744) disk./.usagePercent‚76.01834439897436 disk.dm-0.in‚525.75" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚33615.47802734375& disk.nvme1n1.out‚33615.47802734375$ memory_percent‚97.22416400909424 network.sent‚ 1.9625298e+07$ disk./.usageGB‚1167.071632385254 disk.rbd0.out‚0( disk.nvme0c0n1.out‚68333.77197265625 disk.nvme0n1.in‚ 295.87890625& disk.nvme0n1.out‚68333.77197265625 disk.md0.in‚0( proc.memory.availableMB‚ 454.79296875" disk.nvme0c0n1.in‚ 295.87890625¯öCœì'‚(:‘ ñªöÒ¾Ò¶Ì gpu.0.gpu‚58.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚173.9' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚57.96666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o…kã4í'/ _wandbruntime‚11175  _runtime‚11175æ6î'î'ŒN J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99549836 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚408956110710784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14362036237312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121676172 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14259833764015 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14281383911424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1882230542188544 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99549838 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14362139402240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221236167 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14281383911424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1896512284570624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚423318128335872 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1871217883954368 _runtime‚11175.289807026 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1885477700679023 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121676172 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14281725342720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99549836 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845180017139628e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1882230559227904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121686329 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚408956092098560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1012 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29354884 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221236167 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1882230542188544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 221226008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29355181 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14281725342720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 trainer/global_step‚12299 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50813911()›ð ï'‘N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚423318128335872 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1871217883954368 _runtime‚11175.289807026 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1885477700679023 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121676172 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14281725342720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99549836 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845180017139628e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1882230559227904 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121686329 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚408956092098560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1012 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29354884 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221236167 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1882230542188544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 221226008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes·W%'.all.current‚ 121777152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29355181 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14281725342720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 trainer/global_step‚12299 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50813911 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99549836 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚408956110710784 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14362036237312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121676172 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14259833764015 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14281383911424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1882230542188544 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99549838 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14362139402240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221236167 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14281383911424 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1896512284570624 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1903832194171392 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1889496009832960 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1903831835707904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99931695 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚305 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚410581943944704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21543993 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1889495992793600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1889495992793600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1903831835707904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29466653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14336184338432 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1903832194171392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14416872681472 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1878441986660496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1892756535810941 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1878441969621136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122161311 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424998816626176 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222082861 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122151168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99931695 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122161311 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222093006 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51010646 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424998690647552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122151168 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚11219.880603292 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99931693 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14314208589171 _step‚1015 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99931693 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21543996 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14335842914304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14416765315072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1889496009832960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚410581925332480 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0êÀ¸Î (ü'†P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1889495992793600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1889495992793600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1903831835707904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29466653 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14336184338432 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1903832194171392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14416872681472 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1878441986660496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1892756535810941 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1878441969621136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122161311 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424998816626176 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222082861 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122151168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99931695 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122161311 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222093006 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51010646 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424998690647552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122151168 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚11219.880603292 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99931693 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14314208589171 _step‚1015 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99931693 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21543996 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14335842914304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14416765315072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1889496009832960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚410581925332480 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14335842914304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.784518046304991e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚308 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 trainer/global_step‚12349 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51010954 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1903832194171392 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1889496009832960 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1903831835707904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99931695 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚305 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚410581943944704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21543993÷.¤žqôý'‚(:æ ž«öÒ¨áòË# disk.dm-1.out‚14453.30224609375 disk.nvme0n1.in‚ 295.87890625% disk.nvme1c1n1.out‚33618.86328125 disk.nvme1n1.in‚ 229.87109375# disk.nvme1n1.out‚33618.86328125( proc.memory.availableMB‚ 455.01953125 cpu‚25.69450070854949 network.sent‚ 1.973427e+07 disk.rbd0.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1167.2387046813965 disk.md0.in‚0 disk.dm-0.in‚525.75 disk.dm-0.out‚86338.75390625 disk.dm-1.in‚0" disk.nv%9xÂ}me0c0n1.in‚ 295.87890625& disk.nvme0n1.out‚68393.75244140625$ memory_percent‚97.22278118133545 network.recv‚ 2.94502e+06) disk./.usagePercent‚76.02922681526765 disk.md0.out‚ 586.953125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚68393.75244140625" disk.nvme1c1n1.in‚ 229.87109375 proc.memory.rssMB‚ 2950.90625* proc.memory.percent‚18.010902404785156«wÑ^žþ'‚(:“ ž«öÒ©²ÄÌ gpu.0.gpu‚67.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.412' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.13733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸’‡4ÿ'/ _wandbruntime‚11220  _runtime‚11220žâç'€(‹N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1889644284335104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21545730 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1892904717266031 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14336953013248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14315657538809 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99939486 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222100310 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚11220.508634002 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29469252 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51014688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚410615105602560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1892905074866777 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14315316977423 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222110469 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122160824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚410615124214784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1903981220308992 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1903981578779648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29468961 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122170981 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222110469 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1903981578779648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14417981715456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222100310 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1889644267295744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 $ _timestamp‚1.784518046932838e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚425032984153088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14417878550528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51014688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚410615105602560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1892905074866777 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14315316977423 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222110469 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122160824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚410615124214784 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1903981220308992 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1903981578779648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29468961 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122170981 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222110469 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1903981578779648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14417981715456 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222100310 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1889644267295744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 $ _timestamp‚1.784518046932838e+09 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚425032984153088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14417878550528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚11220.508634002 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29469252 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180ø‚¾õ 4‚(/ _wandbruntime‚11220  _runtime‚11220+ºÉšƒ(ˆ 0loss/FRt2/assignment_loss‚0.6613650378519601 2loss/SRqqt2/assignment_loss‚0.5095999810283888 3loss/FBt2/assignment_loss‚0.0024850266417670466 /loss/FRt2/detection_loss‚1.4654852709961814 1loss/SRqqt2/detection_loss‚0.5906700705941289 epoch‚30 _runtime‚11220.509569614 /loss/FBt1/detection_loss‚0.4765233997369823 $loss/total_loss‚7.14329328223104 $ _timestamp‚1.784518046934067e+09 _step‚1017 3loss/FBt1/assignment_loss‚0.0001339558955230039 /loss/FRt1/detection_loss‚0.4762947103987355 trainer/global_step‚12349 3loss/SRqqt1/assignment_loss‚0.39279625191662926 1loss/SRqqt1/detection_loss‚0.6419212261144739 /loss/FBt2/detection_loss‚0.4614208631986833 0loss/FRt1/assignment_loss‚1.4645974878575865yâvù„( 1loss/SRqqt1/detection_loss‚0.6419212261144739 /loss/FBt2/detection_loss‚0.4614208631986833 0loss/FRt1/assignment_loss‚1.4645974878575865 0loss/FRt2/assignment_loss‚0.6613650378519601 2loss/SRqqt2/assignment_loss‚0.5095999810283888 3loss/FBt2/assignment_loss‚0.0024850266417670466 /loss/FRt2/detection_loss‚1.4654852709961814 1loss/SRqqt2/detection_loss‚0.5906700705941289 epoch‚30 _runtime‚11220.509569614 /loss/FBt1/detection_loss‚0.4765233997369823 $loss/total_loss‚7.14329328223104 $ _timestamp‚1.784518046934067e+09 _step‚1017 3loss/FBt1/assignment_loss‚0.0001339558955230039 /loss/FRt1/detection_loss‚0.4762947103987355 trainer/global_step‚12349 3loss/SRqqt1/assignment_loss‚0.39279625191662926ùD?W¥p…(‚(:å ­«öÒ¼ÙãË disk.nvme1n1.in‚ 229.87109375# disk.nvme1n1.out‚33619.79296875 disk.md0.out‚587.375# disk.dm-1.out‚14453.40380859375' disk.nvme0c0n1.out‚68579.0966796875$ memory_percent‚97.22151756286621) disk./.usagePercent‚76.06185890723276 disk.dm-0.in‚525.75# disk.dm-0.out‚86524.00439453125 disk.dm-1.in‚0" disk.nvme1c1n1.in‚ 229.87109375 proc.memory.rssMB‚ 2950.90625 cpu‚25.694730781421622 proc.cpu.threads‚17 disk.rbd0.in‚0 network.sent‚ 1.9785014e+07$ disk./.usageGB‚1167.739688873291 disk.md0.in‚0" disk.nvme0c0n1.in‚ 295.87890625 disk.nvme0n1.in‚ 295.87890625% disk.nvme0n1.out‚68579.0966796875' proc.memory.availableMB‚ 455.2265625% disk.nvme1c1n1.out‚33619.79296875* proc.memory.percent‚18.010902404785156 disk.rbd0.out‚0 network.recv‚ 2.949552e+06.¬¶ž†(‚(:“ ­«öÒæ÷Ì gpu.0.gpu‚91.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚292.343' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.44766666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05 w‡(‚(:ì ¼«öÒèÿÑË& disk.nvme0c0n1.out‚69628.650390625 disk.nvme0n1.in‚ 295.87890625 disk.nvme1n1.in‚ 229.87109375# disk.dm-0.out‚87710.22998046875 cpu‚25.694982546750122" disk.nvme0c0n1.in‚ 295.87890625" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 587.703125$ memory_percent‚97.22154140472412* proc.memory.percent‚18.010902404785156 network.sent‚ 1.9826195e+07 disk.dm-1.in‚0" disk.dm-1.out‚14453.4716796875( disk.nvme1c1n1.out‚33757.24365234375& disk.nvme1n1.out‚33757.24365234375 disk.dm-0.in‚525.75( proc.memory.availableMB‚ 455.22265625 proc.memory.rssMB‚ 2950.90625 network.recv‚ 2.953606e+06) disk./.usagePercent‚76.20630455296623$ disk./.usageGB‚1169.957290649414$ disk.nvme0n1.out‚69628.650390625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0›s1žˆ(‚(:“ ¼«öÒöÝœÌ gpu.0.gpu‚62.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.908' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.30266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öª_´4‰(/ _wandbruntime‚11263  _runtime‚11263é0©þÉŠ(à trainer/global_step‚12399 $ _timestamp‚1.784518090177857e+09 _runtime‚11263.755083913 _step‚1018 & lr-AdamW/pg1‚0.0002011793425754841 & lr-AdamW/pg2‚0.0002011793425754841×<›Î‹(È & lr-AdamW/pg1‚0.0002011793425754841 & lr-AdamW/pg2‚0.0002011793425754841 trainer/global_step‚12399 $ _timestamp‚1.784518090177857e+09 _runtime‚11263.755083913 _step‚1018ú«›xà4Œ(/ _wandbruntime‚11263  _runtime‚11263Ä89Çœ(…P UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426713654763008 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1885813520034096 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1885813502994736 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122635820 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100321345 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1896909717900800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1896909734940160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29581514 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222967308 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51211942 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚426713532978688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412240957448704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚412240938836480 _runtime‚11263.755170413 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100321345 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1911301488380416 JDeviceStatsMonitor.on_trž½5o ain_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29581249 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100321343 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚12399 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1911301129916928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122635820 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1896909734940160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14370032363853 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784518090178516e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14391753440256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21630696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14472697314304 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122645963 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1900183552397949 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _step‚1019 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360ý¡#(Ž(ŠP XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426713654763008 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1885813520034096 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1885813502994736 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122635820 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100321345 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1896909717900800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1896909734940160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29581514 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222967308 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51211942 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚426713532978688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412240957448704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚412240938836480 _runtime‚11263.755170413 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100321345 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1911301488380416 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29581249 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100321343 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚12399 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1911301129916928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122635820 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1896909734940160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14370032363853 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784518090178516e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14391753440256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21630696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14472697314304 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122645963 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1900183552397949 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _step‚1019 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0û¤šÉ’4(/ _wandbruntime‚11264  _runtime‚11264οˆ«'(N PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚11264.368806065 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1885960950701568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51215966 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122655633 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21632430 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1911450872988672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14473707088896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14392863546368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51216242 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14392522115072 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1897057992402944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122655633 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412274119106560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412274137718784 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1885960933662208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222974612 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222984771 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1020 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122645476 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14473812350976 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1900331733853039 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1897058009442304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122645476 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1900332091453785 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29583812 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222984771 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3åÁ<¢$‘(•N SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚11264.368806065 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1885960950701568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51215966 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122655633 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21632430 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1911450872988672 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14473707088896 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14392863546368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51216242 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14392522115072 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1897057992402944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122655633 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412274119106560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412274137718784 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1885960933662208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222974612 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222984771 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1020 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122645476 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14473812350976 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1900331733853039 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1897058009442304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122645476 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1900332091453785 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29583812 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 QCDeviceStatsMonDOÓéitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222984771 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0üß)X44’(/ _wandbruntime‚11264  _runtime‚11264ðÆpY“(ˆ %loss/total_loss‚7.546642214616737 _step‚1021 1loss/FBt1/assignment_loss‚0.02274202479422988 /loss/FRt1/detection_loss‚0.4908407499763517 0loss/FBt1/detection_loss‚0.42554232832349714 trainer/global_step‚12399 % _timestamp‚1.7845180907940598e+09 3loss/FBt2/assignment_loss‚0.0013032222730199041 /loss/FRt2/detection_loss‚1.5512893869382367 1loss/SRqqt1/detection_loss‚0.7441807999005038 epoch‚30 2loss/SRqqt2/assignment_loss‚0.4859236715233041 0loss/FBt2/detection_loss‚0.42463217874589776 _runtime‚11264.369391826 /loss/FRt1/assignment_loss‚1.555229004018016 0loss/FRt2/assignment_loss‚0.7434571596163497 2loss/SRqqt1/assignment_loss‚0.4437978365069186 1loss/SRqqt2/detection_loss‚0.6577038520004116ý®¿”( /loss/FRt1/detection_loss‚0.4908407499763517 0loss/FBt1/detection_loss‚0.42554232832349714 trainer/global_step‚12399 % _timestamp‚1.7845180907940598e+09 3loss/FBt2/assignment_loss‚0.0013032222730199041 /loss/FRt2/detection_loss‚1.5512893869382367 1loss/SRqqt1/detection_loss‚0.7441807999005038 epoch‚30 2loss/SRqqt2/assignment_loss‚0.4859236715233041 0loss/FBt2/detection_loss‚0.42463217874589776 _runtime‚11264.369391826 /loss/FRt1/assignment_loss‚1.555229004018016 0loss/FRt2/assignment_loss‚0.7434571596163497 2loss/SRqqt1/assignment_loss‚0.4437978365069186 1loss/SRqqt2/detection_loss‚0.6577038520004116 %loss/total_loss‚7.546642214616737 _step‚1021 1loss/FBt1/assignment_loss‚0.02274202479422988ý.5z•(‚(:ï Ë«öÒØ‚¤Ë disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 295.87890625% proc.memory.availableMB‚ 451.15625* proc.memory.percent‚18.035006523132324" disk.dm-1.out‚14453.5224609375 network.recv‚ 2.957233e+06% disk./.usageGB‚1167.2355117797852 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚33921.88916015625 disk.md0.out‚ 588.06640625 disk.dm-0.in‚525.75# memory_percent‚97.2463607788086 network.sent‚ 1.9829159e+07( disk.nvme1c1n1.out‚33921.88916015625 disk.md0.in‚0( disk.nvme0c0n1.out‚71032.47509765625& disk.nvme0n1.out‚71032.47509765625# proc.memory.rssMB‚ 2954.85546875 cpu‚25.6948136456337 proc.cpu.threads‚17" disk.dm-0.out‚89277.8505859375 disk.dm-1.in‚0 disk.nvme0n1.in‚ 295.87890625) disk./.usagePercent‚76.02901884251979" disk.nvme1c1n1.in‚ 229.87109375 disk.rbd0.in‚0?0ÞZž–(‚(:“ Ë«öÒ䮽Í gpu.0.gpu‚98.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚196.872' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.62400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qr·x—(‚(:í Ú«öÒà÷¹Ë) disk./.usagePercent‚76.02897486261972% disk./.usageGB‚1167.2348365783691 disk.dm-0.in‚525.75 disk.nvme0n1.in‚ 295.87890625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 588.30859375# proc.memory.rssMB‚ 2979.87109375 network.recv‚ 2.96145e+06 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 295.87890625& disk.nvme0n1.out‚71035.76123046875" disk.nvme1c1n1.in‚ 229.87109375$ disk.nvme1n1.out‚33922.388671875 disk.dm-0.out‚ 89280.9609375( disk.nvme0c0n1.out‚71035.76123046875& disk.nvme1c1n1.out‚33922.388671875' proc.memory.availableMB‚ 426.0078125 cpu‚25.697984498674955 network.sent‚ 1.9893732e+07# disk.dm-1.out‚14453.66650390625$ memory_percent‚97.39985466003418* proc.memory.percent‚18.187689781188965 proc.cpu.threads‚17‘Õ˜(‚(:’ Ú«öÒª¿þË gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.384' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.46133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z)4™(/ _wandbruntime‚11280  _runtime‚11280È¥àÚ=š(—{ )loss/val_total_loss‚7.496597761527129 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972329139709473 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6295965909957886 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.996094286441803 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6090301275253296 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_3_of_3‚0.9994120597839355 jet/accuracy_2_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚12399 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23863869905471802 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6737132668495178 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3689938187599182 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6509501338005066 particle/accuracy_1_of_1‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30660319328308105 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08265520632266998 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5895856022834778 particle/accuracy_5_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6789495348930359 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08324194699525833 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7900014519691467 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 'particle/f_score‚0.6622521281242371 _runtime‚11280.25577575 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1272348165512085 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7235304117202759 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9118664264678955 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 particle/accuracy_2_of_3‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.390625 % _timestamp‚1.7845181066796668e+09 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_3‚0.9283719658851624 +jet/accuracy_4_of_4‚0.01670061983168125 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947232604026794 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8190288543701172 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967668652534485 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8989607691764832 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9085034132003784 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.31589648127555847 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2935958802700043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3274328410625458 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2831074893474579 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8940854668617249 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 epoch‚30 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17500001192092896 *jet/accuracy_1_of_4‚0.9708527326583862 /particle/accuracy_4_of_4‚0.9572455883026123 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08190222829580307 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08103059977293015 *jet/accuracy_2_of_4‚0.7831095457077026 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7638888955116272 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2816525995731354 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8982992768287659 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04481314867734909 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6523598432540894 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08286550641059875 particle/accuracy_2_of_2‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2485865205526352 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08286550641059875 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2926013469696045 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6649478673934937 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6494277119636536 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4685848653316498 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3265286982059479 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32493194937705994 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.650062620639801 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 jet/accuracy_4_of_5‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8989607691764832 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6552777290344238 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9895342588424683 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.31561926007270813 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6244814395904541 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 +jet/accuracy_3_of_3‚0.08010473102331161 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36881405115127563 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9868274927139282 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 'particle/accuracy‚0.856476366519928 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6796875 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998410165309906 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7638888955116272 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6544421911239624 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21673978865146637 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3247307240962982 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.991936445236206 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6087210774421692 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6087210774421692 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5624999403953552 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9582587480545044 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7223966717720032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2284688651561737 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7226194143295288 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.609737753868103 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4804871380329132 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3263772428035736 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.31561926007270813 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23901741206645966 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6737502217292786 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6838577389717102 *jet/accuracy_1_of_2‚0.7923958897590637 jet/accuracy_5_of_5‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7437246441841125 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10986161231994629 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2271009385585785 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5964560508728027 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6253078579902649 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09842034429311752 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6766584515571594 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3600620627403259 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.48372453451156616 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_1_of_1‚0.34986209869384766 +jet/accuracy_2_of_2‚0.16628164052963257 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 _step‚1022 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5821014046669006 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6293261647224426 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36363834142684937 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6278762221336365 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6985903978347778 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 jet/accuracy_1_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2704019248485565 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6668069362640381 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6491772532463074 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4677078127861023 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28511282801628113 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.055414874106645584 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7238953113555908 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5279802083969116 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6760460734367371 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2152755856513977 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 5 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.30096235871315 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7900014519691467 (particle/specificity‚0.9114990234375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6937559247016907 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.607205867767334 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.31957635283470154 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9957318902015686 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49480560421943665 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7009700536727905 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.33826717734336853 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05677013099193573 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2893918454647064 4validation_average_jet_accuracy‚0.38897705078125 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7389903664588928 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06412247568368912 jet/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6532652378082275 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7660220861434937 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.987604022026062 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2193482220172882 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7278396487236023 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976621270179749 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8774339556694031 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8950017094612122 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_2_of_3‚0.5723212361335754eÉÓØY.›(œ{ =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7660220861434937 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.987604022026062 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2193482220172882 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7278396487236023 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976621270179749 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8774339556694031 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8950017094612122 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 *jet/accuracy_2_of_3‚0.5723212361335754 )loss/val_total_loss‚7.496597761527129 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972329139709473 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6295965909957886 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.996094286441803 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6090301275253296 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_3_of_3‚0.9994120597839355 jet/accuracy_2_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚12399 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23863869905471802 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6737132668495178 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3689938187599182 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6509501338005066 particle/accuracy_1_of_1‚1 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.30660319328308105 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08265520632266998 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5895856022834778 particle/accuracy_5_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6789495348930359 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08324194699525833 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7900014519691467 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 'particle/f_score‚0.6622521281242371 _runtime‚11280.25577575 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1272348165512085 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7235304117202759 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9118664264678955 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 particle/accuracy_2_of_3‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.390625 % _timestamp‚1.7845181066796668e+09 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_1_of_3‚0.9283719658851624 +jet/accuracy_4_of_4‚0.01670061983168125 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947232604026794 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8190288543701172 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967668652534485 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8989607691764832 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9085034132003784 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.31589648127555847 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2935958802700043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3274328410625458 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2831074893474579 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8940854668617249 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 epoch‚30 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.17500001192092896 *jet/accuracy_1_of_4‚0.9708527326583862 /particle/accuracy_4_of_4‚0.9572455883026123 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08190222829580307 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08103059977293015 *jet/accuracy_2_of_4‚0.7831095457077026 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7638888955116272 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2816525995731354 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8982992768287659 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04481314867734909 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6523598432540894 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08286550641059875 particle/accuracy_2_of_2‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2485865205526352 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08286550641059875 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2926013469696045 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6649478673934937 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6494277119636536 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4685848653316498 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3265286982059479 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.32493194937705994 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.650062620639801 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 jet/accuracy_4_of_5‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8989607691764832 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6552777290344238 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9895342588424683 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.31561926007270813 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6244814395904541 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 +jet/accuracy_3_of_3‚0.08010473102331161 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36881405115127563 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9868274927139282 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 'particle/accuracy‚0.856476366519928 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6796875 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998410165309906 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7638888955116272 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6544421911239624 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21673978865146637 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3247307240962982 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.991936445236206 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6087210774421692 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6087210774421692 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5624999403953552 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9582587480545044 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7223966717720032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2284688651561737 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7226194143295288 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.609737753868103 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4804871380329132 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3263772428035736 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.31561926007270813 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23901741206645966 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6737502217292786 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6838577389717102 *jet/accuracy_1_of_2‚0.7923958897590637 jet/accuracy_5_of_5‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7437246441841125 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10986161231994629 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2271009385585785 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 9"Pur! 2tIity/0FRt*SRqqt0FBt/event_purity‚0.5964560508728027 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6253078579902649 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09842034429311752 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6766584515571594 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3600620627403259 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.48372453451156616 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_1_of_1‚0.34986209869384766 +jet/accuracy_2_of_2‚0.16628164052963257 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 _step‚1022 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5821014046669006 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6293261647224426 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36363834142684937 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6278762221336365 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6985903978347778 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 jet/accuracy_1_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2704019248485565 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6668069362640381 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6491772532463074 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4677078127861023 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28511282801628113 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.055414874106645584 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7238953113555908 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5279802083969116 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6760460734367371 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2152755856513977 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 5 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.30096235871315 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7900014519691467 (particle/specificity‚0.9114990234375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6937559247016907 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.607205867767334 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.31957635283470154 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9957318902015686 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49480560421943665 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.7009700536727905 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.33826717734336853 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05677013099193573 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2893918454647064 4validation_average_jet_accuracy‚0.38897705078125 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7389903664588928 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06412247568368912 jet/accuracy_3_of_5‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6532652378082275þÚE«itœ(‚(:é é«öÒä¯ÊË network.sent‚ 1.9896755e+07# disk.dm-1.out‚14453.73681640625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚71045.88525390625 disk.nvme1n1.in‚ 229.87109375* proc.memory.percent‚18.195724487304688 proc.cpu.threads‚17 network.recv‚ 2.965111e+06) disk./.usagePercent‚76.02896020265302 disk.dm-0.in‚525.75 disk.dm-1.in‚0" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1c1n1.out‚33924.830078125# memory_percent‚97.4069595336914% proc.memory.availableMB‚ 424.84375% disk./.usageGB‚1167.2346115112305 disk.md0.in‚0 disk.md0.out‚ 588.6640625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 295.87890625& disk.nvme0n1.out‚71045.88525390625$ disk.nvme1n1.out‚33924.830078125 proc.memory.rssMB‚ 2981.1875" disk.dm-0.out‚89292.6826171875" disk.nvme0c0n1.in‚ 295.87890625 cpu‚25.697853946925836f[Éž(‚(:“ é«öÒ©õ¶Ì gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚257.272' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.75733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]—W·sž(‚(:è ø«öÒÒ¬µË( disk.nvme1c1n1.out‚33925.37646484375 disk.md0.in‚0 disk.md0.out‚ 589.015625 disk.dm-1.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚76.02898380768414% disk.nvme0n1.out‚71047.6611328125" disk.nvme1c1n1.in‚ 229.87109375# disk.dm-0.out‚89294.18505859375% proc.memory.availableMB‚ 424.84375* proc.memory.percent‚18.195724487304688 cpu‚25.69752647319698& disk.nvme1n1.out‚33925.37646484375 disk.dm-0.in‚525.75 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 295.87890625' disk.nvme0c0n1.out‚71047.6611328125 proc.memory.rssMB‚ 2981.1875 network.sent‚ 1.9955657e+07 network.recv‚ 2.969323e+06 disk.nvme1n1.in‚ 229.87109375! disk.dm-1.out‚14453.798828125# memory_percent‚97.4069595336914 proc.cpu.threads‚17% disk./.usageGB‚1167.2349739074707 disk.nvme0n1.in‚ 295.87890625:"çŸ(‚(:’ ø«öÒʽÌ gpu.0.gpu‚92.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚266.39' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.79666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡¼jø4 (/ _wandbruntime‚11322  _runtime‚11322L³·ÔË¡(Å ' lr-AdamW/pg1‚0.00019887180782202213 ' lr-AdamW/pg2‚0.00019887180782202213 trainer/global_step‚12449 % _timestamp‚1.7845181487287564e+09 _runtime‚11322.30603633 _step‚1023½âТ(Ê ' lr-AdamW/pg1‚0.00019887180782202213 ' lr-AdamW/pg2‚0.00019887180782202213 trainer/global_step‚12449 % _timestamp‚1.7845181487287564e+09 _runtime‚11322.30603633 _step‚1023ÿU®X}4£(/ _wandbruntime‚11322  _runtime‚11322£ëçl (¤(…P A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14549200885760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21721509 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21721512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100833801 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚428471731888640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚12449 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14467298953728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29730833 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚11322.30609848 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51452342 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123212109 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1906265786883584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14467640377856 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1920733085837312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29731124„…ø(¥(ŠP F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100833803 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123201966 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100833803 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123201966 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚428471851575808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14549301960704 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1920733444300800 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1920733444300800 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1906265803922944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚413922549615104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚413922531002880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14445478864884 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1909551771241444 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123212109 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _step‚1024 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14467298953728 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1906265786883584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51452342 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123212109 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1906265786883584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14467640377856 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1920733085837312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29731124 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14549200885760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21721509 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21721512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100833801 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚428471731888640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚12449 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14467298953728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29730833 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚11322.30609848 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1906414061385728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428506031673344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1906414061385728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100841596 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14468409052672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51456373 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11322.916152633 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1906414078425088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14550415176704 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1909700310297168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14468750483968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14446927814522 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224053216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1895253723044032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1909700667897914 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_eîÀ-pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1920882470438400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224063375 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21723243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224063375 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1906414078425088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14550320400384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 % _timestamp‚1.7845181493404176e+09 _step‚1025 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1909700310297168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14468750483968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14446927814522 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224053216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1895253723044032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1909700667897914 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1920882470438400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224063375 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21723243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224063375 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1906414078425088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14550320400384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 % _timestamp‚1.7845181493404176e+09 _step‚1025 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1906414061385728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428506031673344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1906414061385728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100841596 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14468409052672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51456373 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11322.916152633ïÌÕ4©(/ _wandbruntime‚11322  _runtime‚11322ìª;Yª(‰ 0loss/FRt2/assignment_loss‚0.6973679051095364 2loss/SRqqt1/assignment_loss‚0.4490970007790065 /loss/FRt1/detection_loss‚0.4490807812193388 &loss/total_loss‚7.4441891234057795 epoch‚31 _step‚1026 3loss/SRqqt2/assignment_loss‚0.45386606235620375 3loss/FBt1/assignment_loss‚0.0017041498257720279 2loss/FBt2/assignment_loss‚0.016942575785960774 1loss/SRqqt1/detection_loss‚0.6110095620679955 /loss/FBt1/detection_loss‚0.4315766133799722 /loss/FRt2/detection_loss‚1.5178394996419586 /loss/FBt2/detection_loss‚0.5411901463359956 $ _timestamp‚1.784518149342883e+09 0loss/FRt1/assignment_loss‚1.5662361337929909 1loss/SRqqt2/detection_loss‚0.7082786931110485 trainer/global_step‚12449 _runtime‚11322.918207937 ¨`«(Ž 0loss/FRt2/assignment_loss‚0.6973679051095364 2loss/SRqqt1/assignment_loss‚0.4490970007790065 /loss/FRt1/detection_loss‚0.4490807812193388 &loss/total_loss‚7.4441891234057795 epoch‚31 _step‚1026 3loss/SRqqt2/assignment_loss‚0.45386606235620375 3loss/FBt1/assignment_loss‚0.0017041498257720279 2loss/FBt2/assignment_loss‚0.016942575785960774 1loss/SRqqt1/detection_loss‚0.6110095620679955 /loss/FBt1/detection_loss‚0.4315766133799722 /loss/FRt2/detection_loss‚1.5178394996419586 /loss/FBt2/detection_loss‚0.5411901463359956 $ _timestamp‚1.784518149342883e+09 0loss/FRt1/assignment_loss‚1.5662361337929909 1loss/SRqqt2/detection_loss‚0.7082786931110485 trainer/global_step‚12449 _runtime‚11322.918207937‚ÕKAÒw¬(‚(:ì ‡¬öÒÏíË disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375* proc.memory.percent‚18.195724487304688 proc.cpu.threads‚17 network.recv‚ 2.972832e+06 disk.md0.in‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚72964.2900390625 disk.nvme0n1.in‚ 295.87890625( disk.nvme1c1n1.out‚33926.10986328125( proc.memory.availableMB‚ 424.80859375 proc.memory.rssMB‚ 2981.1875 network.sent‚ 1.9958579e+07' disk./.usagePercent‚76.167756543255% disk./.usageGB‚1169.3654823303223# disk.dm-0.out‚91210.81494140625" disk.nvme0c0n1.in‚ 295.87890625 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚33926.10986328125 cpu‚25.697177924143052 disk.md0.out‚ 589.30859375! disk.dm-1.out‚14453.869140625% disk.nvme0n1.out‚72964.2900390625# memory_percent‚97.4071741104126 disk.dm-0.in‚525.75 disk.rbd0.in‚0Áîž­(‚(:“ ‡¬öÒ¨½¶Ï gpu.0.gpu‚56.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚288.357' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.11900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–Nr*w®(‚(:ì –¬öÒíœÛË# disk.dm-0.out‚91211.40087890625" disk.nvme0c0n1.in‚ 295.87890625 disk.nvme0n1.in‚ 295.87890625" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚33926.64697265625 disk.nvme1n1.in‚ 229.87109375$ memory_percent‚97.40889072418213( proc.memory.availableMB‚ 424.52734375) disk./.usagePercent‚76.20887029561172' disk.nvme0c0n1.out‚72965.0595703125 disk.md0.out‚ 589.5703125 proc.cpu.threads‚17 network.sent‚ 2.0023027e+07 network.recv‚ 2.976699e+06& disk.nvme1n1.out‚33926.64697265625 proc.memory.rssMB‚ 2981.1875* proc.memory.percent‚18.195724487304688 cpu‚25.696739093102224$ disk./.usageGB‚1169.996681213379 disk.dm-0.in‚525.75 disk.dm-1.in‚0 disk.dm-1.out‚14454.01953125 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚72965.0595703125 disk.md0.in‚0P⓯(‚(:ˆ –¬öÒëë©Ì gpu.0.gpu‚57.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.473' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.491 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µÙõÛr°(‚(:ç ¥¬öÒÔÀ¾Ë" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme0n1.in‚ 295.87890625% disk.nvme0n1.out‚72970.4052734375% disk.nvme1n1.out‚33927.5361328125! disk.dm-0.out‚91216.376953125' disk.nvme0c0n1.out‚72970.4052734375$ memory_percent‚97.40955829620361 proc.memory.rssMB‚ 2981.1875 cpu‚25.696414686268135 proc.cpu.threads‚17' disk.nvme1c1n1.out‚33927.5361328125 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 590.1171875 disk.dm-1.out‚ 14454.109375 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 424.41796875* proc.memory.percent‚18.195724487304688 network.sent‚ 2.0025949e+07 network.recv‚ 2.980339e+06) disk./.usagePercent‚76.17791863338816% disk./.usageGB‚1169.5214958190918 disk.md0.in‚0 disk.dm-0.in‚525.75 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 295.87890625ᇯ¤ž±(‚(:“ ¥¬öÒƯ›Ì gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚267.503' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.16766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ʪ…4²(/ _wandbruntime‚11364  _runtime‚11364ã7üTȳ( _step‚1027 % lr-AdamW/pg1‚0.000196571233657616 % lr-AdamW/pg2‚0.000196571233657616 trainer/global_step‚12499 % _timestamp‚1.7845181909140565e+09 _runtime‚11364.489620224^ÕšòÍ´(Ç % lr-AdamW/pg1‚0.000196571233657616 % lr-AdamW/pg2‚0.000196571233657616 trainer/global_step‚12499 % _timestamp‚1.7845181909140565e+09 _runtime‚11364.489620224 _step‚1027ƒðz*ž4µ(/ _wandbruntime‚11364  _runtime‚11364'g‡È (¶(…P ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430186714238464 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123686618 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101223453 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1913679511990784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101223453 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1913679529030144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14501302639566 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14522868055552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101223451 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224910069 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123696761 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚12499 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14523209479680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101223451 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107104256 _runtime‚11364.490406716 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1913679511990784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1928202380046336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14522868055552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415581544506880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14523209479680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430186607134208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1913679529030144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14605062627328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1928202738509824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21808209 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1916979145429086 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224920214 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14500962078292 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51653999 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29845790 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 % _timestamp‚1.7845181909149559e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14605151119360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1902477842789520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51654270 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415581563119104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1928202738509824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1928202380046336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101223451 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107104256 _runtime‚11364.490406716 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1913679511990784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1928202380046336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14522868055552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415581544506880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14523209479680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430186607134208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1913679529030144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14605062627328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1928202738509824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21808209 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1916979145429086 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224920214 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14500962078292 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51653999 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29845790 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 % _timestamp‚1.7845181909149559e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14605151119360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1902477842789520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51654270 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚415581563119104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1928202738509824 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1928202380046336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647„x{’Å4¸(/ _wandbruntime‚11365  _runtime‚11365¾ š'¹(N UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1913827803532288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29848290 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224937677 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14502070466544 trainer/global_step‚12499 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123706431 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430221015546880 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101231246 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1913827786492928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21809943 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚11365.100310252 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123696274 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415614724776960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123706431 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14523978154496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14606181575680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14524319585792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚430220906352640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1902625273456992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224937677 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224927518 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224927518 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1913827786492928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123696274 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1902625256417632 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101231244 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1917127684484922 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21809946 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101231244 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1928352123118080 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1913827803532288 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29848017 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109194240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14606272157696 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51657960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101231246 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1928351764647424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415614743389184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14524319585792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51658236 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1917127326884176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1928352123118080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0žvæ<'º(”N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14524319585792 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚430220906352640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1902625273456992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224937677 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 224927518 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224927518 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1913827786492928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123696274 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1902625256417632 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101231244 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1917127684484922 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21809946 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101231244 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1928352123118080 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1913827803532288 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29848017 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109194240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14606272157696 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51657960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101231246 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1928351764647424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415614743389184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14524319585792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51658236 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1917127326884176 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1928352123118080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1913827803532288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29848290 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224937677 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14502070466544 trainer/global_step‚12499 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123706431 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430221015546880 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101231246 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1913827786492928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21809943 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚11365.100310252 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123696274 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415614724776960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123706431 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14523978154496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14606181575680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344…Œi @4»(/ _wandbruntime‚11365  _runtime‚11365aßì¼(‹ %loss/total_loss‚7.251773663870162 epoch‚31 2loss/SRqqt2/assignment_loss‚0.5395856789759138 3loss/FBt1/assignment_loss‚0.0034190347324387603 3loss/FBt2/assignment_loss‚0.0015495543387270701 0loss/FRt1/detection_loss‚0.47646926155546127 /loss/FRt2/detection_loss‚1.5407172609333888 1loss/SRqqt1/detection_loss‚0.6312050157010828 /loss/FBt1/detection_loss‚0.4428654623817104 0loss/FBt2/detection_loss‚0.34360507400889023 trainer/global_step‚12499 _runtime‚11365.103391648 1loss/SRqqt2/detection_loss‚0.6554970042533742 0loss/FRt2/assignment_loss‚0.5967435422744459 2loss/SRqqt1/assignment_loss‚0.4385246693080085 0loss/FRt1/assignment_loss‚1.5815921054067206 % _timestamp‚1.7845181915280256e+09 _step‚1030_ØbŠ½( _runtime‚11365.103391648 1loss/SRqqt2/detection_loss‚0.6554970042533742 0loss/FRt2/assignment_loss‚0.5967435422744459 2loss/SRqqt1/assignment_loss‚0.4385246693080085 0loss/FRt1/assignment_loss‚1.5815921054067206 % _timestamp‚1.7845181915280256e+09 _step‚1030 %loss/total_loss‚7.251773663870162 epoch‚31 2loss/SRqqt2/assignment_loss‚0.5395856789759138 3loss/FBt1/assignment_loss‚0.0034190347324387603 3loss/FBt2/assignment_loss‚0.0015495543387270701 0loss/FRt1/detection_loss‚0.47646926155546127 /loss/FRt2/detection_loss‚1.5407172609333888 1loss/SRqqt1/detection_loss‚0.6312050157010828 /loss/FBt1/detection_loss‚0.4428654623817104 0loss/FBt2/detection_loss‚0.34360507400889023 trainer/global_step‚12499†ýT«hx¾(‚(:í ´¬öÒŠý€Ì disk.md0.out‚ 590.33203125) disk./.usagePercent‚76.17488128706836% disk./.usageGB‚1169.4748649597168& disk.nvme0n1.out‚73708.79345703125" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚34056.8896484375% disk.nvme1n1.out‚34056.8896484375 disk.rbd0.out‚0 network.recv‚ 2.983789e+06 disk.nvme0n1.in‚ 295.87890625" disk.nvme0c0n1.in‚ 295.87890625( disk.nvme0c0n1.out‚73708.79345703125( proc.memory.availableMB‚ 424.13671875 proc.memory.rssMB‚ 2981.1875 proc.cpu.threads‚17 disk.dm-0.in‚525.75 network.sent‚ 2.0044757e+07 disk.nvme1n1.in‚ 229.87109375 disk.rbd0.in‚0* proc.memory.percent‚18.195724487304688 cpu‚25.6960656326317 disk.md0.in‚0! disk.dm-0.out‚92083.537109375 disk.dm-1.in‚0# disk.dm-1.out‚14454.21142578125$ memory_percent‚97.41127490997314F]‚Ÿ¿(‚(:” ´¬öÒ›ý½Ì gpu.0.gpu‚43.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚176.541' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚58.846999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d˜þvÀ(‚(:ë ìöÒ ´±Ë% disk.nvme1n1.out‚34059.4111328125 disk.md0.in‚0 disk.md0.out‚ 590.90625 network.sent‚ 2.0093306e+07' disk.nvme0c0n1.out‚73722.1748046875 disk.nvme0n1.in‚ 295.87890625' disk.nvme1c1n1.out‚34059.4111328125 disk.dm-0.in‚525.75 network.recv‚ 2.988148e+06# disk.dm-1.out‚14454.28173828125( proc.memory.availableMB‚ 424.38671875 proc.memory.rssMB‚ 2981.1875 cpu‚25.695744836711295 proc.cpu.threads‚17) disk./.usagePercent‚76.17487929927626% disk./.usageGB‚1169.4748344421387" disk.nvme0c0n1.in‚ 295.87890625% disk.nvme0n1.out‚73722.1748046875# disk.dm-0.out‚92098.14697265625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375# memory_percent‚97.4097490310669* proc.memory.percent‚18.195724487304688¿JS+ Á(‚(:‘RY@ ìöÒ•œîË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.824' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.27466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M=£†yÂ(‚(:î Ò¬öÒû½ÎË network.sent‚ 2.0100254e+07) disk./.usagePercent‚76.17487706301014 disk.dm-1.in‚0! disk.dm-1.out‚14454.466796875" disk.nvme0c0n1.in‚ 295.87890625' disk.nvme0c0n1.out‚73723.6728515625 disk.nvme1n1.in‚ 229.87109375* proc.memory.percent‚18.195724487304688% disk./.usageGB‚1169.4748001098633 disk.md0.out‚ 591.48828125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 295.87890625( disk.nvme1c1n1.out‚34060.26513671875& disk.nvme1n1.out‚34060.26513671875 cpu‚25.69539898011739 disk.md0.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚73723.6728515625 proc.memory.rssMB‚ 2981.1875 network.recv‚ 2.991485e+06 disk.dm-0.in‚525.75# disk.dm-0.out‚92099.08544921875" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.40993976593018( proc.memory.availableMB‚ 424.35546875 proc.cpu.threads‚17ÜF9“Ã(‚(:ˆ Ò¬öÒÍŸ‘Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚122.379' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.793 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þŸO`4Ä(/ _wandbruntime‚11406  _runtime‚11406÷—ÚuÌÅ(Æ _step‚1031 ' lr-AdamW/pg1‚0.00019427776784384541 ' lr-AdamW/pg2‚0.00019427776784384541 trainer/global_step‚12549 % _timestamp‚1.7845182332742023e+09 _runtime‚11406.849740057ý§—cÑÆ(Ë ' lr-AdamW/pg2‚0.00019427776784384541 trainer/global_step‚12549 % _timestamp‚1.7845182332742023e+09 _runtime‚11406.849740057 _step‚1031 ' lr-AdamW/pg1‚0.00019427776784384541‡ }å4Ç(/ _wandbruntime‚11406  _runtime‚11406òøø~ (È(†P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101613103 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14578778581504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14556445291700 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚11406.851522041 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225784371 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124181413 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14578778581504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21894909 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1935672032718848 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124181413 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1924405804415460 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 % _timestamp‚1.7845182332748756e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101613101 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1924406162016094 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1921093237097984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1935671674255360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14660978494976 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101613103 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51855659 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚1032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225794516 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1909849376163120 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1935672032718848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚431901424945152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14578437157376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14578437157376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29960750 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431901555118080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51855960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚12549 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚417240558010880¤æol(É(‹P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124181413 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14578778581504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21894909 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1935672032718848 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124181413 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1924405804415460 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 % _timestamp‚1.7845182332748756e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101613101 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1924406162016094 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1921093237097984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1935671674255360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14660978494976 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101613103 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51855659 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚1032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225794516 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1909849376163120 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1935672032718848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚431901424945152 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14578437157376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14578437157376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29960750 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431901555118080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51855960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚12549 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚417240558010880 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101613103 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14578778581504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14556445291700 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚11406.851522041 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225784371ˆ‚ŒÅ¾4Ê(/ _wandbruntime‚11407  _runtime‚11407Nï‹;Á&Ë(N D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431935720515072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124191083 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1921241528639488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21896646 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚417273756893184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11407.461930052 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1924554701071930 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101620894 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1921241511600128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7845182338861578e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 225801820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124191083 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚417273738280960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚1033 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124180926 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14579547256320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1909996806830592 trainer/global_step‚12549 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101620894 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29963294 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1935821058856448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51859940 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14579888687616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14557553679952 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14579547256320 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29963018 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1921241511600128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225811979 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51859661 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431935850680832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101620896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21896643 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14557894241338 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14661982234112 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1924554343471184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1935821417327104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101620896 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14579547256320 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29963018 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1921241511600128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225811979 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51859661 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431935850680832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101620896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21896643 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14557894241338 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14661982234112 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1924554343471184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1935821417327104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101620896 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431935720515072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124191083 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1921241528639488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21896646 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚417273756893184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11407.461930052 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1924554701071930 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101620894 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1921241511600128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7845182338861578e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 225801820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124191083 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚417273738280960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚1033 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124180926 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14579547256320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1909996806830592 trainer/global_step‚12549 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101620894 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29963294 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1935821058856448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51859940 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14579888687616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14557553679952 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360‰¸\î¥4Í(/ _wandbruntime‚11407  _runtime‚114077ñ, Î(† _step‚1034 0loss/FRt2/assignment_loss‚0.6979097611073292 3loss/SRqqt1/assignment_loss‚0.46690091108220644 2loss/SRqqt2/assignment_loss‚0.4996519874667678 /loss/FRt1/assignment_loss‚1.587920730046823 1loss/SRqqt2/detection_loss‚0.6034838109006898 $loss/total_loss‚7.45787495347221 epoch‚31 0loss/FRt1/detection_loss‚0.42709574863797506 1loss/SRqqt1/detection_loss‚0.6449440889601737 /loss/FBt2/detection_loss‚0.5231264554008487 2loss/FBt1/assignment_loss‚0.004173168508564165 1loss/FBt2/assignment_loss‚0.00083851474784289 /loss/FRt2/detection_loss‚1.5294873222703997 0loss/FBt1/detection_loss‚0.47234245434258904 trainer/global_step‚12549 $ _timestamp‚1.784518233889342e+09 _runtime‚11407.464656097ïíÏ(‹ /loss/FBt2/detection_loss‚0.5231264554008487 2loss/FBt1/assignment_loss‚0.004173168508564165 1loss/FBt2/assignment_loss‚0.00083851474784289 /loss/FRt2/detection_loss‚1.5294873222703997 0loss/FBt1/detection_loss‚0.47234245434258904 trainer/global_step‚12549 $ _timestamp‚1.784518233889342e+09 _runtime‚11407.464656097 _step‚1034 0loss/FRt2/assignment_loss‚0.6979097611073292 3loss/SRqqt1/assignment_loss‚0.46690091108220644 2loss/SRqqt2/assignment_loss‚0.4996519874667678 /loss/FRt1/assignment_loss‚1.587920730046823 1loss/SRqqt2/detection_loss‚0.6034838109006898 $loss/total_loss‚7.45787495347221 epoch‚31 0loss/FRt1/detection_loss‚0.42709574863797506 1loss/SRqqt1/detection_loss‚0.6449440889601737Š^JÝlgÐ(‚(:Ü á¬öÒʸÒË network.recv‚ 2.99516e+06" disk.nvme1n1.out‚ 34132.6640625 disk.dm-1.in‚0 proc.memory.rssMB‚ 2981.1875& disk.nvme0c0n1.out‚73994.173828125$ disk.nvme0n1.out‚73994.173828125 disk.dm-0.in‚525.75* proc.memory.percent‚18.195724487304688 cpu‚25.69503308436532 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375$ disk.nvme1c1n1.out‚ 34132.6640625 disk.md0.in‚0 disk.md0.out‚ 592.0859375# memory_percent‚97.4116325378418& proc.memory.availableMB‚ 424.078125 proc.cpu.threads‚17) disk./.usagePercent‚76.02886304931442% disk./.usageGB‚1167.2331199645996 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 295.87890625 disk.nvme0n1.in‚ 295.87890625 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.out‚92440.55078125! disk.dm-1.out‚14454.626953125 network.sent‚ 2.0116061e+07qN¬ÕžÑ(‚(:“ á¬öÒ‰´Ì gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚266.361' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.78699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ÊzvÒ(‚(:ë ð¬öÒ«àÁË disk.rbd0.out‚0( proc.memory.availableMB‚ 424.32421875 network.sent‚ 2.016754e+07 network.recv‚ 2.999113e+06) disk./.usagePercent‚76.02889162382577 disk.dm-1.out‚14454.69140625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 592.30859375 disk.dm-0.out‚92606.27734375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚74077.75341796875( disk.nvme1c1n1.out‚34215.35498046875& disk.nvme1n1.out‚34215.35498046875 disk.dm-0.in‚525.75 proc.memory.rssMB‚ 2981.1875* proc.memory.percent‚18.195724487304688 cpu‚25.69473443272463" disk.nvme1c1n1.in‚ 229.87109375% disk./.usageGB‚1167.2335586547852" disk.nvme0c0n1.in‚ 295.87890625 disk.nvme0n1.in‚ 295.87890625& disk.nvme0n1.out‚74077.75341796875$ memory_percent‚97.41013050079346 proc.cpu.threads‚17 disk.rbd0.in‚0ð57+‘Ó(‚(:† ð¬öÒº¥ŒÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.269' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.423 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸòãLiÔ(‚(:Þ ÿ¬öÒËžÁË proc.cpu.threads‚17 network.recv‚ 3.002527e+06 disk.dm-0.in‚525.75 disk.dm-1.out‚ 14454.765625 disk.rbd0.out‚0 proc.memory.rssMB‚ 2981.1875 network.sent‚ 2.0170504e+07) disk./.usagePercent‚76.02884963171779 disk.nvme0n1.in‚ 295.87890625% disk.nvme0n1.out‚74087.1865234375 disk.dm-1.in‚0' disk.nvme0c0n1.out‚74087.1865234375$ memory_percent‚97.41015434265137 cpu‚25.69447959396673" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚34217.1318359375% disk.nvme1n1.out‚34217.1318359375 disk.md0.in‚0 disk.md0.out‚592.6875 disk.rbd0.in‚0' proc.memory.availableMB‚ 424.3203125* proc.memory.percent‚18.195724487304688% disk./.usageGB‚1167.2329139709473 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.out‚ 92616.59375" disk.nvme0c0n1.in‚ 295.87890625_ÄBÕ(‚(:’ ÿ¬öÒüë•Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.13' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.37666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù»ÐÞ4Ö(/ _wandbruntime‚11449  _runtime‚11449-Ò­kÌ×(Æ ' lr-AdamW/pg1‚0.00019199155768573383 ' lr-AdamW/pg2‚0.00019199155768573383 trainer/global_step‚12599 % _timestamp‚1.7845182756930573e+09 _runtime‚11449.268737657 _step‚1035Ô˜ÑØ(Ë trainer/global_step‚12599 % _timestamp‚1.7845182756930573e+09 _runtime‚11449.268737657 _step‚1035 ' lr-AdamW/pg1‚0.00019199155768573383 ' lr-AdamW/pg2‚0.00019199155768573383‹>è¹"4Ù(/ _wandbruntime‚11449  _runtime‚11449¹BÇ (Ú(†P HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52058178 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚433616302238208 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1931833178603102 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226658673 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124666065 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1931832821002468 % _timestamp‚1.7845182756941297e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1928506962205184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1917220909536720 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚12599 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1943141326927872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124666065 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124655922 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1943141326927872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1928506979244544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102002751 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1928506979244544 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21981609 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14716730723328 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _step‚1036 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226668818 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1943140968464384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21981612 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚433616428216832 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1917220892497360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102002751 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1943140968464384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _runtime‚11449.271426893 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124655922 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30076274 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14634347683328 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14634006259200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418899590127104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1928506962205184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1943140968464384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21981612 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚433616428216832 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1917220892497360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102002751 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1943140968464384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _runtime‚11449.271426893 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124655922 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30076274 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceS$C¿ltatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14634347683328 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14634006259200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418899590127104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1928506962205184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102010544 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52062233 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433650722124800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52061931 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚433650594056192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14717951727616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14613036893360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1943290353065472 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚418932770397184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1943290711536128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14635116358144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1928655236707328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30078887 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124665578 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1917368323164832 _step‚1037 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226676122 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418932751784960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102010544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30078588 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14613377454746 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226686281 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124675735 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226686281 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14635457789440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1943290711536128 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102010546 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21983346 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102010546œÛB'Þ(•N \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1928655253746688 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚11449.879057156 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1928655236707328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21983343 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14717842271232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1917368340204192 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102010544 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52062233 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433650722124800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52061931 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚433650594056192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14717951727616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14613036893360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1943290353065472 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚418932770397184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1943290711536128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14635116358144 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1928655236707328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚302 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30078887 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124665578 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1917368323164832 _step‚1037 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226676122 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418932751784960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102010544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30078588 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14613377454746 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226686281 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124675735 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226686281 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14635457789440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1943290711536128 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102010546 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21983346 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102010546 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14635457789440 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124675735 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚420558603631104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 227532975 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1950610262673408 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1935920704351744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14772571335680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30191267 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1950610621136896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 227543120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batYÏ¥Æch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1935920687312384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1950610621136896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102392403 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14689575361024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1935920687312384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125140574 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102392401 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125150717 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435331156354560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1939259837589476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1924592442910320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1950610262673408 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14689575361024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1939260195190110 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1935920704351744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14667411718516 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 $ _timestamp‚1.784518318382234e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102392401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1950610262673408 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14689575361024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1939260195190110 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1935920704351744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14667411718516 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 $ _timestamp‚1.784518318382234e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102392401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚420558603631104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 227532975 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1950610262673408 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1935920704351744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14772571335680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30191267 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1950610621136896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 227543120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1935920687312384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1950610621136896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102392403 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14689575361024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1935920687312384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125140574 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102392401 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125150717 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435331156354560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1939259837589476 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1924592442910320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0™ÇÔ‰4î(/ _wandbruntime‚11492  _runtime‚11492€¢½'ï(N bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14773793150464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52263522 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _runtime‚11492.568301988 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚420591783901184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚420591765288960 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1924739856538432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14668520106768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1950759647274496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52263802 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1939408376645200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14691026891264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30193479 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1939408734245946 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102400194 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1924739873577792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14668860668154 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102400194 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125150230 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1950760005745152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚435365451080192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1936068978853888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102400196 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125160387 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227560583 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14691026891264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227560583 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227550424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7845183189925394e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125160387 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 _step‚1041 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435365577051648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1950760005745152 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30193756 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚12649 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2לC½ð(•N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102400194 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125150230 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1950760005745152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚435365451080192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1936068978853888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102400196 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125160387 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227560583 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14691026891264 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227560583 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227550424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7845183189925394e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 12ÕÂÏQ^5160387 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 _step‚1041 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435365577051648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1950760005745152 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30193756 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚12649 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14773793150464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52263522 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _runtime‚11492.568301988 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚420591783901184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚420591765288960 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1924739856538432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14668520106768 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1950759647274496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52263802 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1939408376645200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14691026891264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30193479 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1939408734245946 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102400194 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1924739873577792 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14668860668154‘…•I`4ñ(/ _wandbruntime‚11492  _runtime‚11492ÎUÂÆò(Š 0loss/FBt1/detection_loss‚0.42689083807547185 epoch‚31 3loss/FBt2/assignment_loss‚0.0012765003591252118 .loss/FRt2/detection_loss‚1.524254389386406 1loss/SRqqt2/detection_loss‚0.7049391239158416 0loss/FBt2/detection_loss‚0.45513883642548614 trainer/global_step‚12649 % _timestamp‚1.7845183189950066e+09 _runtime‚11492.570283339 _step‚1042 0loss/FRt1/assignment_loss‚1.5808686051792369 2loss/SRqqt2/assignment_loss‚0.4738598267161587 3loss/FBt1/assignment_loss‚0.0008159341319808278 /loss/FRt1/detection_loss‚0.4638129409375114 %loss/total_loss‚7.414229455022822 2loss/SRqqt1/assignment_loss‚0.4376911043059213 1loss/SRqqt1/detection_loss‚0.6280212294183902 0loss/FRt2/assignment_loss‚0.7166601261712907"r¡ó( 1loss/SRqqt2/detection_loss‚0.7049391239158416 0loss/FBt2/detection_loss‚0.45513883642548614 trainer/global_step‚12649 % _timestamp‚1.7845183189950066e+09 _runtime‚11492.570283339 _step‚1042 0loss/FRt1/assignment_loss‚1.5808686051792369 2loss/SRqqt2/assignment_loss‚0.4738598267161587 3loss/FBt1/assignment_loss‚0.0008159341319808278 /loss/FRt1/detection_loss‚0.4638129409375114 %loss/total_loss‚7.414229455022822 2loss/SRqqt1/assignment_loss‚0.4376911043059213 1loss/SRqqt1/detection_loss‚0.6280212294183902 0loss/FRt2/assignment_loss‚0.7166601261712907 0loss/FBt1/detection_loss‚0.42689083807547185 epoch‚31 3loss/FBt2/assignment_loss‚0.0012765003591252118 .loss/FRt2/detection_loss‚1.524254389386406’©bÌžô(‚(:“ »­öÒËœ¹Ë gpu.0.gpu‚92.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚279.644' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.21466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~}D-rõ(‚(:ç »­öÒÈ‚²Ì$ disk.nvme0n1.out‚76774.939453125# disk.nvme1n1.out‚34348.34765625 disk.md0.in‚0 disk.md0.out‚ 594.4296875# disk.dm-0.out‚95431.38134765625 disk.dm-1.in‚0# disk.dm-1.out‚14455.21142578125" disk.nvme1c1n1.in‚ 229.87109375% disk.nvme1c1n1.out‚34348.34765625 disk.nvme1n1.in‚ 229.87109375 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚76774.939453125 proc.memory.rssMB‚ 2981.1875* proc.memory.percent‚18.195724487304688% disk./.usageGB‚1169.7111473083496 disk.dm-0.in‚525.75" disk.nvme0c0n1.in‚ 295.87890625$ memory_percent‚97.40941524505615 cpu‚25.69313716338494 proc.cpu.threads‚17 network.sent‚ 2.0259511e+07 network.recv‚ 3.015949e+06) disk./.usagePercent‚76.19027176735709( proc.memory.availableMB‚ 424.44140625 disk.nvme0n1.in‚ 295.87890625†9FXtö(‚(:é Ê­öÒ¤ÂÄË# disk.dm-1.out‚14455.30419921875 disk.rbd0.in‚0 cpu‚25.692803350425034 disk.dm-1.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚76775.6259765625* proc.memory.percent‚18.195724487304688) disk./.usagePercent‚76.19026853719492 disk.md0.out‚ 594.55078125# disk.dm-0.out‚95432.01025390625" disk.nvme0c0n1.in‚ 295.87890625' disk.nvme0c0n1.out‚76775.6259765625" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375$ memory_percent‚97.40791320800781 network.recv‚ 3.019765e+06 disk.nvme0n1.in‚ 295.87890625& disk.nvme1c1n1.out‚34348.662109375$ disk.nvme1n1.out‚34348.662109375$ proc.memory.availableMB‚424.6875 proc.memory.rssMB‚ 2981.1875 proc.cpu.threads‚17 network.sent‚ 2.0311247e+07% disk./.usageGB‚1169.7110977172852 disk.md0.in‚0 disk.dm-0.in‚525.75ø®ÄÒ÷(‚(:’ Ê­öÒ¨ÿË gpu.0.gpu‚71.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚291.95' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.31666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑÂ&¥4ø(/ _wandbruntime‚11534  _runtime‚11534Å…øîÈù( & lr-AdamW/pg1‚0.0001874414912170647 & lr-AdamW/pg2‚0.0001874414912170647 trainer/global_step‚12699 # _timestamp‚1.78451836085911e+09 _runtime‚11534.436012788 _step‚1043Pg!ùÍú(Ç _step‚1043 & lr-AdamW/pg1‚0.0001874414912170647 & lr-AdamW/pg2‚0.0001874414912170647 trainer/global_step‚12699 # _timestamp‚1.78451836085911e+09 _runtime‚11534.436012788“%Ð 4û(/ _wandbruntime‚11534  _runtime‚11534Ÿšˆ (ü(…P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125625226 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _step‚1044 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30305359 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚12699 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102782051 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437045994205184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52460368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1958079556882432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125635369 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102782051 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1931963959244560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30305632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437046107600896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22155012 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1946687211777118 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1931963976283920 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1958079915345920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1943334412419584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7845183608597527e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1943334429458944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚422217598522880 _runtime‚11534.436074708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52460644 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1958079556882432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22155009 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14745485886976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534PjIN(ý(ŠP XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30305632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437046107600896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22155012 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1946687211777118 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1931963976283920 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1958079915345920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1943334412419584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7845183608597527e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1943334429458944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚422217598522880 _runtime‚11534.436074708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52460644 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1958079556882432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22155009 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14745485886976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125625226 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _step‚1044 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30305359 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚12699 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102782051 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437045994205184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52460368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1958079556882432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125635369 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102782051 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1931963959244560”$ãѱwþ(‚(:ì Ù­öÒÙ¿ÆË% disk.nvme1n1.out‚34350.8623046875 disk.md0.in‚0 disk.md0.out‚ 594.9296875 disk.rbd0.out‚0& disk.nvme0n1.out‚76790.08447265625 network.sent‚ 2.0327012e+07# disk.dm-1.out‚14455.35498046875' proc.memory.availableMB‚ 424.4453125 proc.memory.rssMB‚ 2981.1875* proc.memory.percent‚18.195724487304688 proc.cpu.threads‚17 network.recv‚ 3.022857e+06" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 295.87890625 disk.nvme0n1.in‚ 295.87890625$ disk./.usageGB‚1169.473289489746 disk.dm-0.in‚525.75 disk.dm-1.in‚0( disk.nvme0c0n1.out‚76790.08447265625$ memory_percent‚97.40939140319824 cpu‚25.692423056134498" disk.dm-0.out‚95447.8037109375( disk./.usagePercent‚76.1747786673015' disk.nvme1c1n1.out‚34350.8623046875À›÷žÿ(‚(:“ Ù­öÒšûÖÌ gpu.0.gpu‚92.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚253.575' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.52499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îDG4€)/ _wandbruntime‚11535  _runtime‚11535ÁÒÝêI)N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14746254561792 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1958229299954176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14746254561792 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228434885 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 228424726 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125645039 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102789844 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14829506547712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 % _timestamp‚1.7845183614694293e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14746595993088 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14724343881562 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1932111389912032 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125634882 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1943482686921728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102789846 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚12699 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚422250797405184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22156746 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1946835393232208 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚11535.045058054 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228434885 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52464608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30307581 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14829616004096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1943482703961088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30307862 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚1045 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102789844 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1943482686921728 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1958228941483520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52464324 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1943482703961088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1946835750832954 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437080413409280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228434885 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52464608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30307581 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14829616004096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1943482703961088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30307862 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚1045 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102789844 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1943482686921728 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1958228941483520 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52464324 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1943482703961088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1946835750832954 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437080413409280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14746254561792 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228434885 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 228424726 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125645039 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102789844 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14829506547712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 % _timestamp‚1.7845183614694293e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14746595993088 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14724343881562 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1932111389912032 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125634882 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1943482686921728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102789846 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚12699 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚422250797405184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22156746 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240•àüvÂ4ƒ)/ _wandbruntime‚11535  _runtime‚11535]Ôo„)‹ epoch‚31 _runtime‚11535.047845856 _step‚1046 /loss/FRt1/assignment_loss‚1.490159769120665 3loss/SRqqt1/assignment_loss‚0.41990771069465027 2loss/SRqqt2/assignment_loss‚0.4405653566782141 4loss/FBt1/assignment_loss‚0.00038767372060253527 /loss/FRt2/detection_loss‚1.5281525358601529 1loss/SRqqt1/detection_loss‚0.6455946342999946 0loss/FBt2/detection_loss‚0.45877714505019995 trainer/global_step‚12699 2loss/FBt2/assignment_loss‚0.002321286787949903 %loss/total_loss‚7.155389846103205 % _timestamp‚1.7845183614725907e+09 0loss/FRt2/assignment_loss‚0.6513811494322385 /loss/FRt1/detection_loss‚0.4399456767666029 1loss/SRqqt2/detection_loss‚0.6486499316159995 0loss/FBt1/detection_loss‚0.42954697607593534}…) _step‚1046 /loss/FRt1/assignment_loss‚1.490159769120665 3loss/SRqqt1/assignment_loss‚0.41990771069465027 2loss/SRqqt2/assignment_loss‚0.4405653566782141 4loss/FBt1/assignment_loss‚0.00038767372060253527 /loss/FRt2/detection_loss‚1.5281525358601529 1loss/SRqqt1/detection_loss‚0.6455946342999946 0loss/FBt2/detection_loss‚0.45877714505019995 trainer/global_step‚12699 2loss/FBt2/assignment_loss‚0.002321286787949903 %loss/total_loss‚7.155389846103205 % _timestamp‚1.7845183614725907e+09 0loss/FRt2/assignment_loss‚0.6513811494322385 /loss/FRt1/detection_loss‚0.4399456767666029 1loss/SRqqt2/detection_loss‚0.6486499316159995 0loss/FBt1/detection_loss‚0.42954697607593534 epoch‚31 _runtime‚11535.047845856–@*Nãu†)‚(:ê è­öÒ¨–Ë( disk.nvme1c1n1.out‚34384.18994140625 disk.md0.in‚0" disk.nvme0c0n1.in‚ 295.87890625( disk./.usagePercent‚76.1902931361221" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚76894.7138671875& proc.memory.availableMB‚ 424.640625 disk.dm-1.out‚14455.64453125 network.sent‚ 2.0365946e+07 network.recv‚ 3.02676e+06% disk./.usageGB‚1169.7114753723145 disk.md0.out‚ 595.26171875 disk.dm-0.in‚525.75# disk.dm-0.out‚95584.74658203125 disk.nvme0n1.in‚ 295.87890625& disk.nvme1n1.out‚34384.18994140625% disk.nvme0n1.out‚76894.7138671875$ memory_percent‚97.40819931030273 proc.memory.rssMB‚ 2981.1875* proc.memory.percent‚18.195724487304688 cpu‚25.692110431997786 proc.cpu.threads‚17v2*qž‡)‚(:“ è­öÒˆ½Ì gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚279.442' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.14733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ¥€¢wˆ)‚(:ì ÷­öÒÚ¥ØË disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme1n1.out‚34415.41845703125$ memory_percent‚97.40970134735107( proc.memory.availableMB‚ 424.39453125* proc.memory.percent‚18.195724487304688 cpu‚25.691772859547715) disk./.usagePercent‚76.19029189375203 disk.dm-1.out‚ 14455.703125" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚34415.41845703125 disk.dm-0.in‚525.75 proc.memory.rssMB‚ 2981.1875% disk./.usageGB‚1169.7114562988281 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 295.87890625& disk.nvme0n1.out‚76985.12353515625 proc.cpu.threads‚17 network.sent‚ 2.0420041e+07 network.recv‚ 3.030986e+06( disk.nvme0c0n1.out‚76985.12353515625 disk.nvme0n1.in‚ 295.87890625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 595.55859375! disk.dm-0.out‚95705.666015625ð¥ ó“‰)‚(:ˆ ÷­öÒàÆ™Ì gpu.0.gpu‚71.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚278.235' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó Ð54Š)/ _wandbruntime‚11576  _runtime‚115760Sú=Ê‹)Ä _step‚1047 & lr-AdamW/pg1‚0.0001851779271487737 & lr-AdamW/pg2‚0.0001851779271487737 trainer/global_step‚12749 % _timestamp‚1.7845184029056704e+09 _runtime‚11576.481280177ùÜâÏŒ)É _runtime‚11576.481280177 _step‚1047 & lr-AdamW/pg1‚0.0001851779271487737 & lr-AdamW/pg2‚0.0001851779271487737 trainer/global_step‚12749 % _timestamp‚1.7845184029056704e+09—ÉÙû.4)/ _wandbruntime‚11576  _runtime‚11576âÎá\="Ž)†P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103171701 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1939335509657520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1954113870763492 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103171703 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22241712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1950748154566144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229291724 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126120021 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14800713564672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1965548851091456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52662519 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1965549209554944 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30420807 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14801054988800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1965548851091456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1950748154566144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1950748137526784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1954114228364126 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103171703 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚11576.484982513 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126109878 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14778378145332 trainer/global_step‚12749 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚Ê¥î†Ï 126120021 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126109878 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22241709 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1939335492618160 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1965549209554944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚438761008383488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _step‚1048 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229281579 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528›­Ú()‹P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14801054988800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1965548851091456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1950748154566144 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1950748137526784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1954114228364126 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103171703 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚11576.484982513 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126109878 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14778378145332 trainer/global_step‚12749 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126120021 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126109878 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22241709 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1939335492618160 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1965549209554944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚438761008383488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _step‚1048 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229281579 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103171701 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1939335509657520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1954113870763492 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103171703 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22241712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1950748154566144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229291724 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126120021 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14800713564672 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1965548851091456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52662519 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1965549209554944 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30420807˜ºÙ·`4)/ _wandbruntime‚11577  _runtime‚11577_²0'‘)N ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14779827094970 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229309187 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103179496 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1939482940324992 _runtime‚11577.093102802 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1965698594163200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14802165094912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 $ _timestamp‚1.784518403517357e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229299028 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229309187 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229299028 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1950896412028928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52666540 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1954262767419962 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1954262409819216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚423909792296960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423909810909184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103179496 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14801823663616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126129691 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126129691 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚1049 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1950896412028928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22243443 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22243446 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14779486533584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚438795308841984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 trainer/global_step‚12749 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126119534 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103179494 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1950896429068288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚438795180773376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52666237 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1950896429068288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0*ºú'’)”N C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14779827094970 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229309187 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103179496 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1939482940324992 _runtime‚11577.093102802 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1965698594163200 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14802165094912 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 $ _timestamp‚1.784518403517357e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229299028 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229309187 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 229299028 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1950896412028928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52666540 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1954262767419962 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1954262409819216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚423909792296960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423909810909184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103179496 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14801823663616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126129691 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126129691 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚1049 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1950896412028928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22243443 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22243446 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14779486533584 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚438795308841984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 trainer/global_step‚12749 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126119534 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103179494 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1950896429068288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚438795180773376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52666237 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1950896429068288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303™NË…å4“)/ _wandbruntime‚11577  _runtime‚11577ºÛ·= ”)‡ 0loss/FRt2/assignment_loss‚0.6818291937087324 1loss/SRqqt2/assignment_loss‚0.485637408006836 3loss/FBt1/assignment_loss‚0.0019484402170311936 3loss/FBt2/assignment_loss‚0.0010485372474812848 1loss/SRqqt1/detection_loss‚0.5978277254593675 0loss/SRqqt2/detection_loss‚0.611939911749441 /loss/FBt1/detection_loss‚0.4910229662546527 0loss/FRt1/assignment_loss‚1.4714604673466516 3loss/SRqqt1/assignment_loss‚0.45185713628169066 /loss/FRt2/detection_loss‚1.5106110850596832 /loss/FBt2/detection_loss‚0.6050446299370378 %loss/total_loss‚7.340859550536877 epoch‚31 _step‚1050 /loss/FRt1/detection_loss‚0.4306320492682718 trainer/global_step‚12749 $ _timestamp‚1.784518403519582e+09 _runtime‚11577.094824285Ö‹ÉoD•)Œ /loss/FRt1/detection_loss‚0.4306320492682718 trainer/gßì.£Îlobal_step‚12749 $ _timestamp‚1.784518403519582e+09 _runtime‚11577.094824285 0loss/FRt2/assignment_loss‚0.6818291937087324 1loss/SRqqt2/assignment_loss‚0.485637408006836 3loss/FBt1/assignment_loss‚0.0019484402170311936 3loss/FBt2/assignment_loss‚0.0010485372474812848 1loss/SRqqt1/detection_loss‚0.5978277254593675 0loss/SRqqt2/detection_loss‚0.611939911749441 /loss/FBt1/detection_loss‚0.4910229662546527 0loss/FRt1/assignment_loss‚1.4714604673466516 3loss/SRqqt1/assignment_loss‚0.45185713628169066 /loss/FRt2/detection_loss‚1.5106110850596832 /loss/FBt2/detection_loss‚0.6050446299370378 %loss/total_loss‚7.340859550536877 epoch‚31 _step‚1050šUšku–)‚(:ê †®öÒÚÏ¡Ë# disk.dm-1.out‚14456.53857421875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚76996.8154296875% disk.nvme0n1.out‚76996.8154296875& disk.nvme1c1n1.out‚34422.103515625 proc.memory.rssMB‚ 2981.1875 cpu‚25.69148365931315 network.recv‚ 3.034474e+06' disk./.usagePercent‚76.174789351684 disk.md0.in‚0 disk.nvme1n1.in‚ 229.87109375$ memory_percent‚97.41077423095703 network.sent‚ 2.0423064e+07 disk.md0.out‚ 599.8828125# disk.dm-0.out‚95714.50830078125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 295.8828125" disk.nvme1c1n1.in‚ 229.87109375$ disk.nvme1n1.out‚34422.103515625 proc.cpu.threads‚17% disk./.usageGB‚1169.4734535217285 disk.dm-0.in‚ 525.75390625 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 295.8828125% proc.memory.availableMB‚ 424.21875* proc.memory.percent‚18.195724487304688Ì0Ìü“—)‚(:ˆ †®öÒ‘ñÌ gpu.0.gpu‚85.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚251.238' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚83.746 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é­÷¾t˜)‚(:é •®öÒ‚Ö¬Ë proc.cpu.threads‚17 network.sent‚ 2.0487421e+07$ disk.nvme1n1.out‚34484.478515625 disk.dm-0.in‚ 525.75390625 disk.rbd0.in‚0 network.recv‚ 3.038726e+06) disk./.usagePercent‚76.19027897310342 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 600.2890625 disk.dm-1.in‚0# disk.dm-1.out‚14456.74560546875$ disk.nvme0c0n1.out‚ 77195.6328125* proc.memory.percent‚18.195724487304688% disk./.usageGB‚1169.7112579345703 disk.md0.in‚0# disk.dm-0.out‚95974.62646484375 disk.nvme0n1.in‚ 295.8828125" disk.nvme0n1.out‚ 77195.6328125$ memory_percent‚97.38733768463135 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 295.8828125" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1c1n1.out‚34484.478515625( proc.memory.availableMB‚ 428.05859375 proc.memory.rssMB‚ 2981.1875 cpu‚25.691171294040334ßý ž™)‚(:“ •®öÒ³¶ïË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚126.734' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚42.24466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýˆRÒ~š)‚(:ó ¤®öÒ¨ÈÕË disk.md0.in‚0# disk.dm-0.out‚95978.84619140625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 295.8828125" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.36335277557373) disk./.usagePercent‚76.01618814150027% disk./.usageGB‚1167.0385284423828& disk.nvme1n1.out‚34486.36474609375! disk.dm-1.out‚14457.052734375$ disk.nvme0n1.out‚77200.986328125( disk.nvme1c1n1.out‚34486.36474609375 proc.cpu.threads‚17 network.sent‚ 2.0492351e+07 network.recv‚ 3.042415e+06 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 601.6171875 disk.rbd0.out‚0( proc.memory.availableMB‚ 431.98828125 disk.dm-0.in‚ 525.75390625! disk.nvme0c0n1.in‚ 295.8828125& disk.nvme0c0n1.out‚77200.986328125# proc.memory.rssMB‚ 2989.19921875* proc.memory.percent‚18.244624137878418 cpu‚25.6909016587187469¯:ž›)‚(:“ ¤®öÒÙ€–Ì gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚280.179' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.39299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tpN•4œ)/ _wandbruntime‚11618  _runtime‚11618¶mi’Ë)Å ' lr-AdamW/pg2‚0.00018292220320190533 trainer/global_step‚12799 $ _timestamp‚1.784518445205394e+09 _runtime‚11618.782028184 _step‚1051 ' lr-AdamW/pg1‚0.00018292220320190533Ik"О)Ê ' lr-AdamW/pg1‚0.00018292220320190533 ' lr-AdamW/pg2‚0.00018292220320190533 trainer/global_step‚12799 $ _timestamp‚1.784518445205394e+09 _runtime‚11618.782028184 _step‚1051›€b|4Ÿ)/ _wandbruntime‚11618  _runtime‚11618L€î ( )†P _runtime‚11618.782092704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1961540887350500 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1958161879673344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14856624090624 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1946707043031120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14856624090624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1958161862633984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚12799 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1973018503763968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1973018503763968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230155881 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52864426 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1958161879673344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126604673 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1946707025991760 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52864719 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14834201920014 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1973018145300480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440475745562112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103561351 % _timestamp‚1.7845184452060375e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1973018145300480 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14856282666496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14856282666496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230166026 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1958161862633984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126604673 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22328409 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22328412 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425535625530880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440475873637888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647>3-(¡)‹P N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14856624090624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1958161862633984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚12799 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1973018503763968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1973018503763968 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230155881 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52864426 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1958161879673344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126604673 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1946707025991760 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52864719 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14834201920014 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1973018145300480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440475745562112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103561351 % _timestamp‚1.7845184452060375e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1973018145300480 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14856282666496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14856282666496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230166026 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1958161862633984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126604673 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22328409 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22328412 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425535625530880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440475873637888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _runtime‚11618.782092704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1961540887350500 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1958161879673344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14856624090624 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1946707043031120 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452œaØu4¢)/ _wandbruntime‚11619  _runtime‚11619!õË‘?£)N T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1961689426406224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14941344599040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1958310154175488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 230173330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30538307 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126614343 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230183489 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52868450 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126604186 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126604186 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440510043040768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103569146 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1973167888372224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1973167888372224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1946854473698592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14857392765440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103569144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14834969746992 _runtime‚11619.393268361 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14857734196736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22330143 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425568824413184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14941237239808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230173330 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1958310137136128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚1053 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1961689784006970 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230183489 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19583101541754886áÚ@'¤)•N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52868450 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126604186 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126604186 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440510043040768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103569146 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1973167888372224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1973167888372224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1946854473698592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14857392765440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103569144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14834969746992 _runtime‚11619.393268361 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14857734196736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22330143 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425568824413184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14941237239808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230173330 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1958310137136128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚1053 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1961689784006970 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230183489 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1958310154175488 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1961689426406224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14941344599040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1958310154175488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 230173330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30538307 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126614343 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230183489 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0?±ón4¥)/ _wandbruntime‚11619  _runtime‚11619䩤 ¦)… trainer/global_step‚12799 3loss/SRqqt2/assignment_loss‚0.49273369066077033 0loss/FBt1/detection_loss‚0.40720739600064626 %loss/total_loss‚7.555579343942861 epoch‚31 _runtime‚11619.395727425 /loss/FRt1/assignment_loss‚1.487526771983938 1loss/SRqqt1/assignment_loss‚0.512103561728259 2loss/FBt1/assignment_loss‚0.002502630731968171 0loss/FRt1/detection_loss‚0.45467549499351795 1loss/SRqqt2/detection_loss‚0.6595146202530324 _step‚1054 /loss/FRt2/assignment_loss‚0.729638880448984 /loss/FRt2/detection_loss‚1.5990394426000043 0loss/FBt2/detection_loss‚0.43481453259617253 % _timestamp‚1.7845184458204422e+09 /loss/FBt2/assignment_loss‚0.081365320703883 1loss/SRqqt1/detection_loss‚0.6944570012416847¨ï5§)Š _runtime‚11619.395727425 /loss/FRt1/assignment_loss‚1.487526771983938 1loss/SRqqt1/assignment_loss‚0.512103561728259 2loss/FBt1/assignment_loss‚0.002502630731968171 0loss/FRt1/detection_loss‚0.45467549499351795 1loss/SRqqt2/detection_loss‚0.6595146202530324 _step‚1054 /loss/FRt2/assignment_loss‚0.729638880448984 /loss/FRt2/detection_loss‚1.5990394426000043 0loss/FBt2/detection_loss‚0.43481453259617253 % _timestamp‚1.7845184458204422e+09 /loss/FBt2/assignment_loss‚0.081365320703883 1loss/SRqqt1/detection_loss‚0.6944570012416847 trainer/global_step‚12799 3loss/SRqqt2/assignment_loss‚0.49273369066077033 0loss/FBt1/detection_loss‚0.40720739600064626 %loss/total_loss‚7.555579343942861 epoch‚31ž÷Á©û|¨)‚(:ñ ³®öÒŸÑ–Ë disk.md0.in‚0 disk.dm-0.in‚ 525.75390625# disk.dm-0.out‚95991.65478515625! disk.nvme0c0n1.in‚ 295.8828125" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚34489.02880859375 disk.nvme1n1.in‚ 229.87109375 network.sent‚ 2.0495315e+07% disk./.usageGB‚1167.0384521484375 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 295.8828125& disk.nvme1n1.out‚34489.02880859375 disk.md0.out‚ 602.07421875 disk.rbd0.in‚0& disk.nvme0c0n1.out‚77212.220703125# proc.memory.rssMB‚ 3007.62109375 cpu‚25.691817627008326 network.recv‚ 3.045784e+06! disk.dm-1.out‚14457.166015625$ disk.nvme0n1.out‚77212.220703125$ memory_percent‚97.29347229003906$ proc.memory.availableMB‚443.4375* proc.memory.percent‚18.357062339782715 proc.cpu.threads‚17) disk./.usagePercent‚76.01618317202002Ñ»#“©)‚(:ˆ ³®öÒô¯‰Ì gpu.0.gpu‚67.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.733' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.911 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<A°4ª)/ _wandbruntime‚11635  _runtime‚11635›pJåp=«)êz 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6239960789680481 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6604257225990295 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28894487023353577 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04824680835008621 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968659281730652 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.5003373026847839 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6863940358161926 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31280288100242615 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9121728539466858 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2681295871734619 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6615382432937622 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 +jet/accuracy_3_of_4‚0.09822316467761993 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6853914260864258 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07202813029289246 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7620996832847595 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9130266308784485 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5846157073974609 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_3_of_5‚1 _runtime‚11635.244090678 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29578617215156555 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6256986260414124 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967668652534485 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6634461879730225 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8481671214103699 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17705224454402924 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7092518210411072 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_2_of_2‚0.15751808881759644 *jet/accuracy_2_of_3‚0.5350376963615417 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47795340418815613 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3333328664302826 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33424854278564453 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4322916865348816 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26385655999183655 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2557789981365204 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6913992762565613 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24681775271892548 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2569112479686737 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6354405283927917 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2706368863582611 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6505944728851318 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.18333333730697632 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4281073212623596 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7712036371231079 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.996330738067627 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5813400745391846 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7005208730697632 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6447451710700989 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0778576210141182 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 trainer/global_step‚12799 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.670781135559082 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2885183095932007 particle/accuracy_1_of_5‚1 epoch‚31 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8412510752677917 jet/accuracy_3_of_5‚1 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.946573793888092 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3523596525192261 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18726366758346558 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.058070044964551926 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.604610025882721 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8509070873260498 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4774857461452484 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_1_of_3‚0.9133508801460266 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7083055973052979 *jet/accuracy_1_of_1‚0.3271045982837677 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 particle/accuracy_2_of_5‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03823208436369896 *jet/accuracy_1_of_2‚0.7552157640457153 *jet/accuracy_2_of_4‚0.6939682960510254 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27651768922805786 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5960396528244019 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27688658237457275 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_4_of_4‚0.01670061983168125 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8481671214103699 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_3‚0.07647044956684113 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6905086040496826 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20909717679023743 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07255002856254578 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8855094909667969 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6329016089439392 particle/accuracy_1_of_2‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7005359530448914 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0382683165371418 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 )loss/val_total_loss‚7.461066616057427 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.607546329498291 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6431078910827637 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 _step‚1055 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8643991351127625 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5662968754768372 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6500186324119568 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09316635876893997 particle/accuracy_2_of_2‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24374493956565857 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6860623955726624 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23041556775569916 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25136107206344604 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5813400745391846 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 'particle/f_score‚0.6528343558311462 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24543212354183197 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28841692209243774 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2917732000350952 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19079172611236572 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_4‚0.9710637927055359 7validation_average_jet_accuracy‚0.36644211411476135 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5252912044525146 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34554389119148254 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.996094286441803 particle/accuracy_2_of_4‚1 % _timestamp‚1.7845184616679986e+09 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2011822909116745 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5829100012779236 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7757180333137512 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.058070044964551926 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900868535041809 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972329139709473 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5972253084182739 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44083908200263977 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6802348494529724 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7126753330230713 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9525687098503113 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6858921647071838 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19136130809783936 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19495809078216553 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07881249487400055 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5948560833930969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5878235697746277 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27688658237457275 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5691666603088379 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 particle/accuracy_1_of_1‚1 (particle/accuracy‚0.8537713885307312 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7046797275543213 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7620996832847595 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.884331226348877 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05006104335188866 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6568107008934021 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2196480631828308 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.624693751335144 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6655260324478149 jet/accuracy_1_of_5‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0738382488489151 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9899028539657593 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6402488946914673 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0N+CƬ)ïz 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6500186324119568 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09316635876893997 particle/accuracy_2_of_2‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24374493956565857 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6860623955726624 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23041556775569916 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25136107206344604 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5813400745391846 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 'particle/f_score‚0.6528343558311462 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24543212354183197 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28841692209243774 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2917732000350952 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19079172611236572 & Purity/1FRt2SRqqt1FBt/FBt_purAK.O¯9ity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_4‚0.9710637927055359 7validation_average_jet_accuracy‚0.36644211411476135 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5252912044525146 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34554389119148254 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.996094286441803 particle/accuracy_2_of_4‚1 % _timestamp‚1.7845184616679986e+09 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2011822909116745 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5829100012779236 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7757180333137512 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.058070044964551926 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900868535041809 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972329139709473 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5972253084182739 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44083908200263977 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6802348494529724 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7126753330230713 particle/accuracy_1_of_4‚1 /particle/accuracy_4_of_4‚0.9525687098503113 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6858921647071838 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19136130809783936 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19495809078216553 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07881249487400055 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5948560833930969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5878235697746277 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.27688658237457275 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5691666603088379 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 particle/accuracy_1_of_1‚1 (particle/accuracy‚0.8537713885307312 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7046797275543213 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7620996832847595 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.884331226348877 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05006104335188866 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6568107008934021 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2196480631828308 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.624693751335144 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6655260324478149 jet/accuracy_1_of_5‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0738382488489151 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9899028539657593 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6402488946914673 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6239960789680481 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6604257225990295 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28894487023353577 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04824680835008621 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968659281730652 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.5003373026847839 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6863940358161926 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31280288100242615 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/specificity‚0.9121728539466858 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2681295871734619 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6615382432937622 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 +jet/accuracy_3_of_4‚0.09822316467761993 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6853914260864258 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07202813029289246 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7620996832847595 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9130266308784485 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5846157073974609 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 particle/accuracy_3_of_5‚1 _runtime‚11635.244090678 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29578617215156555 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6256986260414124 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9967668652534485 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6634461879730225 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8481671214103699 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17705224454402924 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7092518210411072 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_2_of_2‚0.15751808881759644 *jet/accuracy_2_of_3‚0.5350376963615417 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47795340418815613 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3333328664302826 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33424854278564453 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4322916865348816 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.26385655999183655 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2557789981365204 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6913992762565613 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24681775271892548 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2569112479686737 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6354405283927917 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2706368863582611 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6505944728851318 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.18333333730697632 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4281073212623596 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7712036371231079 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.996330738067627 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5813400745391846 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7005208730697632 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6447451710700989 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0778576210141182 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 trainer/global_step‚12799 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.670781135559082 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2885183095932007 particle/accuracy_1_of_5‚1 epoch‚31 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8412510752677917 jet/accuracy_3_of_5‚1 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.946573793888092 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3523596525192261 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18726366758346558 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.058070044964551926 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.604610025882721 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8509070873260498 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4774857461452484 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_1_of_3‚0.9133508801460266 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7083055973052979 *jet/accuracy_1_of_1‚0.3271045982837677 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 particle/accuracy_2_of_5‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03823208436369896 *jet/accuracy_1_of_2‚0.7552157640457153 *jet/accuracy_2_of_4‚0.6939682960510254 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27651768922805786 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5960396528244019 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.27688658237457275 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_4_of_4‚0.01670061983168125 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8481671214103699 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_3‚0.07647044956684113 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6905086040496826 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20909717679023743 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07255002856254578 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8855094909667969 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6329016089439392 particle/accuracy_1_of_2‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7005359530448914 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0382683165371418 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 )loss/val_total_loss‚7.461066616057427 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.607546329498291 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.6431078910827637 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 _step‚1055 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8643991351127625 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5662968754768372 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837Ÿ=Ü¡€­)‚(:õ ®öÒÐë¸Ë* proc.memory.percent‚18.413352966308594 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 525.75390625 disk.dm-1.in‚0" disk.dm-1.out‚14457.2958984375 disk.rbd0.out‚0 cpu‚25.693993145270596) disk./.usagePercent‚75.90208216649937 disk.nvme0n1.in‚ 295.8828125# disk.dm-0.out‚95992.19775390625 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 295.8828125$ memory_percent‚97.26483821868896 proc.cpu.threads‚17 network.sent‚ 2.0571705e+07% disk./.usageGB‚1165.2867164611816( disk.nvme1c1n1.out‚34489.51708984375' disk.nvme0c0n1.out‚77213.0439453125 proc.memory.rssMB‚ 3016.84375 network.recv‚ 3.049872e+06% disk.nvme0n1.out‚77213.0439453125" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚34489.51708984375 disk.md0.in‚0 disk.md0.out‚ 602.37109375( proc.memory.availableMB‚ 448.12890625¾‚"Hœ®)‚(:‘ ®öÒ¡ôýË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.672' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.89066666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Òž+v¯)‚(:ë Ñ®öÒ®É–Ë disk.md0.in‚0% disk./.usageGB‚1165.2867736816406 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 295.8828125& disk.nvme0n1.out‚77214.92529296875 disk.md0.out‚ 602.71875) proc.memory.percent‚18.43745708465576 proc.cpu.threads‚17 network.sent‚ 2.0614717e+07) disk./.usagePercent‚75.90208589360955 disk.dm-0.in‚ 525.75390625# memory_percent‚97.1099853515625 cpu‚25.69368080801332 network.recv‚ 3.053919e+06 disk.dm-1.in‚0! disk.dm-0.out‚95993.880859375! proc.memory.availableMB‚473.5# proc.memory.rssMB‚ 3020.79296875" disk.dm-1.out‚14457.3662109375( disk.nvme0c0n1.out‚77214.92529296875 disk.nvme0n1.in‚ 295.8828125" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚34490.12939453125 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚34490.12939453125¯õtž°)‚(:“ Ñ®öÒ¼¼äË gpu.0.gpu‚70.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚289.821' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.60700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á,¦}±)‚(:ò à®öÒ“¿ÜË network.recv‚ 3.057606e+06! disk.dm-1.out‚14457.552734375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 295.8828125" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚34490.68212890625! disk.nvme0c0n1.in‚ 295.8828125( disk.nvme1c1n1.out‚34490.68212890625# memory_percent‚97.0271348953247* proc.memory.percent‚18.453526496887207 cpu‚25.69334411099807( disk.nvme0c0n1.out‚77215.92041015625% disk./.usageGB‚1165.2876052856445 disk.md0.in‚0 disk.md0.out‚ 603.015625! disk.dm-0.out‚95994.572265625# proc.memory.rssMB‚ 3023.42578125( disk./.usagePercent‚75.9021400609441 disk.dm-0.in‚ 525.75390625 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚77215.92041015625 disk.nvme1n1.in‚ 229.87109375( proc.memory.availableMB‚ 487.07421875 proc.cpu.threads‚17 network.sent‚ 2.0621468e+07;öïž²)‚(:“ à®öÒ÷­¦Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚232.333' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚77.44433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔüR4³)/ _wandbruntime‚11676  _runtime‚11676Á ðÌ´)Æ ' lr-AdamW/pg1‚0.00018067446425739332 ' lr-AdamW/pg2‚0.00018067446425739332 trainer/global_step‚12849 % _timestamp‚1.7845185033170686e+09 _runtime‚11676.892682028 _step‚1056Àû&âѵ)Ë ' lr-AdamW/pg2‚0.00018067446425739332 trainer/global_step‚12849 % _timestamp‚1.7845185033170686e+09 _runtime‚11676.892682028 _step‚1056 ' lr-AdamW/pg1‚0.00018067446425739332 ‰Í44¶)/ _wandbruntime‚11676  _runtime‚116767ç·Ø (·)†P G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127170819 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1967517931616768 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1970909463794629 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1955999815373584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127160676 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1967517931616768 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1955999832412944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1982450459684352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14932511028224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30685695 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚12849 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚11676.897069105 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231234485 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚442234016284160 _step‚1057 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30685410 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1982450101220864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1967517948656128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127160676 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231244630 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚427217236309504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127170819 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53104635 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14909988982319 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104073811 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845185033180223e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1982450101220864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚427217217697280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104073809 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15016676802560 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14932511028224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15016779974656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374w‹ Éš ¸)‹P 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231244630 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚427217236309504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127170819 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53104635 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14909988982319 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104073811 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845185033180223e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1982450101220864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚427217217697280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104073809 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15016676802560 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14932511028224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15016779974656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127170819 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1967517931616768 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1970909463794629 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1955999815373584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127160676 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1967517931616768 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1955999832412944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1982450459684352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14932511028224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30685695 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚12849 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚11676.897069105 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231234485 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚442234016284160 _step‚1057 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30685410 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1982450101220864 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1967517948656128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127160676 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1971058002850353 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53108903 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1982599844292608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30687654 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚442268193812992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14910756809297 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104081604 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14933279703040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231251934 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231251934 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1956147246041056 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1982599485821952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427250416579584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1967666223158272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104081602 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1982599844292608 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1967666206118912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231262093 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22420962 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30687941 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53108613 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1971058360451099 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _step‚1058 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127180489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14933279703040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚442268311395840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14933621134336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1967666223158272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14933621134336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 # _timestamp‚1.78451850392584e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚427250397967360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0\Päu'»)’N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53108613 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1971058360451099 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _step‚1058 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127180489 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14933279703040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚442268311395840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14933621134336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1967666223158272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14933621134336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 # _timestamp‚1.78451850392584e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚427250397967360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1971058002850353 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53108903 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1982599844292608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30687654 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚442268193812992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14910756809297 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104081604 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14933279703040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231251934 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231251934 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1956147246041056 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1982599485821952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427250416579584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1967666223158272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104081602 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1982599844292608 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1967666206118912 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231262093 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22420962 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30687941 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488¢;§F4¼)/ _wandbruntime‚11677  _runtime‚11677ŒñÍ+ ½)† /loss/FRt1/assignment_loss‚1.590426931341005 1loss/FBt2/assignment_loss‚0.01593095041158561 /loss/FBt1/detection_loss‚0.5162301717312989 $loss/total_loss‚7.53952732806691 _runtime‚11677.505374682 2loss/SRqqt2/assignment_loss‚0.5174003710280068 0loss/FRt1/detection_loss‚0.46723363479560653 1loss/SRqqt1/detection_loss‚0.6982538397316366 _step‚1059 2loss/SRqqt1/assignment_loss‚0.4776910649446445 3loss/FBt1/assignment_loss‚0.0003979098648673797 1loss/SRqqt2/detection_loss‚0.6563195698916753 epoch‚32 trainer/global_step‚12849 0loss/FRt2/assignment_loss‚0.6883684153488542 /loss/FRt2/detection_loss‚1.5887468813202812 /loss/FBt2/detection_loss‚0.3225275876574478 % _timestamp‚1.7845185039300277e+09‚Ô²H¾)‹ /loss/FRt2/detection_loss‚1.5887468813202812 /loss/FBt2/detection_loss‚0.3225275876574478 % _timestamp‚1.7845185039300277e+09 /loss/FRt1/assignment_loss‚1.590426931341005 1loss/FBt2/assignment_loss‚0.01593095041158561 /loss/FBt1/detection_loss‚0.5162301717312989 $loss/total_loss‚7.53952732806691 _runtime‚11677.505374682 2loss/SRqqt2/assignment_loss‚0.5174003710280068 0loss/FRt1/detection_loss‚0.46723363479560653 1loss/SRqqt1/detection_loss‚0.6982538397316366 _step‚1059 2loss/SRqqt1/assignment_loss‚0.4776910649446445 3loss/FBt1/assignment_loss‚0.0003979098648673797 1loss/SRqqt2/detection_loss‚0.6563195698916753 epoch‚32 trainer/global_step‚12849 0loss/FRt2/assignment_loss‚0.6883684153488542£FPœâ{¿)‚(:ð ï®öÒúç˜Ë disk.md0.in‚0 disk.md0.out‚ 603.40625 disk.dm-1.in‚0% disk./.usageGB‚1165.2877197265625 disk.dm-0.in‚ 525.75390625! disk.dm-1.out‚14457.630859375 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚77218.65380859375" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.00033664703369! disk.dm-0.out‚95997.095703125! disk.nvme0c0n1.in‚ 295.8828125( disk.nvme0c0n1.out‚77218.65380859375( proc.memory.availableMB‚ 491.46484375" proc.memory.rssMB‚ 3024.7421875) proc.memory.percent‚18.46156120300293 disk.nvme0n1.in‚ 295.8828125( disk.nvme1c1n1.out‚34491.38427734375 cpu‚25.6930132306739 proc.cpu.threads‚17 network.sent‚ 2.0624432e+07 network.recv‚ 3.061012e+06) disk./.usagePercent‚75.90214751516446 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚34491.38427734375å`“À)‚(:ˆ ï®öÒü‚¤Ì gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚271.863' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚90.621 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ#ÀÇÁ)‚(:ô þ®öÒ¦ï…Ì network.sent‚ 2.0688793e+07" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-0.in‚ 525.75390625 disk.rbd0.out‚0% disk.nvme0n1.out‚77219.9541015625$ memory_percent‚97.00078964233398& proc.memory.availableMB‚ 491.390625 network.recv‚ 3.065386e+06 disk.md0.out‚ 603.84765625' disk.nvme0c0n1.out‚77219.9541015625) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17% disk./.usageGB‚1165.2874183654785# disk.dm-0.out‚95997.98876953125 disk.dm-1.in‚0# proc.memory.rssMB‚ 3024.74609375 cpu‚25.692698199065326) disk./.usagePercent‚75.90212788571755' disk.nvme1c1n1.out‚34495.2822265625 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚34495.2822265625# disk.dm-1.out‚14460.97314453125 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 295.8828125 disk.nvme0n1.in‚ 295.8828125Õ‹Â)‚(:’ þ®öÒô¨Ë gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚284.91' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.97000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{€£Œ~Ã)‚(:ó ¯öÒ´·©Ë network.sent‚ 2.0691858e+07! disk.dm-0.out‚96000.314453125! disk.nvme0c0n1.in‚ 295.8828125& disk.nvme0n1.out‚77222.51904296875" disk.nvme1c1n1.in‚ 229.87109375$ disk.nvme1n1.out‚34495.998046875' proc.memory.availableMB‚ 491.1484375) proc.memory.percent‚18.46158504486084 network.recv‚ 3.068724e+06) disk./.usagePercent‚75.90213459451586( disk.nvme0c0n1.out‚77222.51904296875 disk.nvme0n1.in‚ 295.8828125& disk.nvme1c1n1.out‚34495.998046875 disk.md0.in‚0 disk.md0.out‚ 604.25390625 cpu‚25.692391186619552 disk.dm-0.in‚ 525.75390625 disk.dm-1.in‚0# disk.dm-1.out‚14461.04345703125 disk.nvme1n1.in‚ 229.87109375$ memory_percent‚97.00226783752441% disk./.usageGB‚1165.2875213623047 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17Z³/«žÄ)‚(:“ ¯öÒ¤°õË gpu.0.gpu‚61.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚291.293' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.09766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(Lv~1Å)/ _wandbruntime‚11718  _runtime‚11r]a718jç ýÊÆ)Ä % _timestamp‚1.7845185452919493e+09 _runtime‚11718.867528472 _step‚1060 & lr-AdamW/pg1‚0.0001784348546833098 & lr-AdamW/pg2‚0.0001784348546833098 trainer/global_step‚12899Ü(jÏÇ)É % _timestamp‚1.7845185452919493e+09 _runtime‚11718.867528472 _step‚1060 & lr-AdamW/pg1‚0.0001784348546833098 & lr-AdamW/pg2‚0.0001784348546833098 trainer/global_step‚12899¤N4ÂÓ4È)/ _wandbruntime‚11718  _runtime‚11718£П (É)†P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127655471 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30800556 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚443948776003072 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15072650071040 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1989919753893376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14987738705920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 % _timestamp‚1.7845185452929318e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127645328 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1974931673763328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232118932 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428876249813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚1061 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15072544801792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1974931673763328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104463459 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30800262 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1963371365786544 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14965472195727 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1989919395429888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127655471 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14988080130048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14988080130048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127645328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚12899 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428876231201280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 232108787 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1978336480381637 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1974931656723968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443948899884544 _runtime‚11718.868367083 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1989919753893376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1978336837982271 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104463461 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53306484 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14987738705920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14965131634453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1989919395429888 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1989919395429888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127655471 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14988080130048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14988080130048 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127645328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚12899 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428876231201280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 232108787 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1978336480381637 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1974931656723968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443948899884544 _runtime‚11718.868367083 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1989919753893376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1978336837982271 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104463461 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53306484 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14987738705920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14965131634453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1989919395429888 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1974931673763328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232118932 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428876249813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚1061 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15072544801792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1974931673763328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104463459 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30800262 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1963371365786544 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14965472195727 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127654984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22507659 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14989190236160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104471254 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104471252 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53310530 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14966580584091 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15073659495424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14988848804864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53310238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127665141 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1978485377038107 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1975079948265472 trainer/global_step‚12899 _step‚1062 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1975079931226112 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232136395 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14989190236160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22507662 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 232126236 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1975079931226112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428909430083584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845185459028764e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14988848804864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127654984 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14966240022705 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1975079948265472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240Ø…aû‹Í)”N EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127654984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22507659 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14989190236160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104471254 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104471252 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53310530 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14966580584091 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15073659495424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14988848804864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53310238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127665141 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1978485377038107 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1975079948265472 trainer/global_step‚12899 _step‚1062 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1975079931226112 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232136395 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14989190236160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22507662 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 232126236 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1975079931226112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428909430083584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845185459028764e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14988848804864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127654984 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14966240022705 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1975079948265472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240¦ï¤ ê4Î)/ _wandbruntime‚11719  _runtime‚11719ÏOj_Ï)ˆ 3loss/FBt2/assignment_loss‚0.0007921572706118342 /loss/FRt2/detection_loss‚1.5235164514527706 %loss/total_loss‚7.659114888173056 $ _timestamp‚1.784518545905761e+09 0loss/FRt1/assignment_loss‚1.6194625124478295 _step‚1063 epoch‚32 0loss/SRqqt2/assignment_loss‚0.50468148496763 3loss/FBt1/assignment_loss‚0.0012670555429684874 1loss/SRqqt1/detection_loss‚0.6967619376400149 0loss/FBt1/detection_loss‚0.41558143612739396 _runtime‚11719.481116565 2loss/SRqqt1/assignment_loss‚0.4490952351165219 0loss/FRt1/detection_loss‚0.47769038567327876 1loss/SRqqt2/detection_loss‚0.6825364811639659 /loss/FBt2/detection_loss‚0.5317037236610693 trainer/global_step‚12899 0loss/FRt2/assignment_loss‚0.7560260271090008ÛPÓêÐ) _runtime‚11719.481116565 2loss/SRqqt1/assignment_loss‚0.4490952351165219 0loss/FRt1/detection_loss‚0.47769038567327876 1loss/SRqqt2/detection_loss‚0.6825364811639659 /loss/FBt2/detection_loss‚0.5317037236610693 trainer/global_step‚12899 0loss/FRt2/assignment_loss‚0.7560260271090008 3loss/FBt2/assignment_loss‚0.0007921572706118342 /loss/FRt2/detection_loss‚1.5235164514527706 %loss/total_loss‚7.659114888173056 $ _timestamp‚1.784518545905761e+09 0loss/FRt1/assignment_loss‚1.6194625124478295 _step‚1063 epoch‚32 0loss/SRqqt2/assignment_loss‚0.50468148496763 3loss/FBt1/assignment_loss‚0.0012670555429684874 1loss/SRqqt1/detection_loss‚0.6967619376400149 0loss/FBt1/detection_loss‚0.41558143612739396§¬Û÷ zÑ)‚(:ï œ¯öÒ”¡•Ë' disk.nvme0c0n1.out‚77223.4443359375 disk.nvme0n1.in‚ 295.8828125 network.recv‚ 3.071327e+06 disk.dm-0.in‚ 525.75390625' proc.memory.availableMB‚ 491.0859375) proc.memory.percent‚18.46158504486084 cpu‚25.692103659739203% disk./.usageGB‚1165.2875022888184" disk.nvme1c1n1.in‚ 229.87109375% disk.nvme1n1.out‚34496.6650390625! disk.dm-0.out‚96000.861328125 disk.dm-1.in‚0 disk.dm-1.out‚14461.26171875 disk.rbd0.in‚0# proc.memory.rssMB‚ 3024.74609375 network.sent‚ 2.0697641e+07 disk.md0.in‚0% disk.nvme0n1.out‚77223.4443359375$ memory_percent‚97.00264930725098 proc.cpu.threads‚17( disk./.usagePercent‚75.9021333521458' disk.nvme1c1n1.out‚34496.6650390625 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 604.63671875 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 295.8828125i )žÒ)‚(:“ œ¯öÒìÕþË gpu.0.gpu‚97.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚274.519' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.50633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{h¬}Ó)‚(:ò «¯öÒö¶Ë% disk./.usageGB‚1163.5769577026367 disk.dm-1.out‚ 14461.3203125! disk.nvme0c0n1.in‚ 295.8828125% disk.nvme0n1.out‚77247.2431640625" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚34497.58935546875' proc.memory.availableMB‚ 491.3359375 network.sent‚ 2.0760363e+07 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme1c1n1.out‚34497.58935546875 disk.nvme1n1.in‚ 229.87109375 cpu‚25.691836200537224 disk.rbd0.in‚0' disk.nvme0c0n1.out‚77247.2431640625 disk.nvme0n1.in‚ 295.8828125 disk.md0.in‚0 disk.dm-0.in‚ 525.75390625# disk.dm-0.out‚96024.84130859375 network.recv‚ 3.075441e+06) disk./.usagePercent‚75.79071536900418 disk.md0.out‚ 604.9453125$ memory_percent‚97.00112342834473# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17özzÙžÔ)‚(:“ «¯öÒ—ÑùË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚131.261' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚43.75366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ß4mtÕ)‚(:é º¯öÒåãÕË disk.md0.in‚0 disk.dm-1.in‚0# disk.dm-1.out‚14461.48193359375" disk.nvme1c1n1.in‚ 229.87109375% disk.nvme1c1n1.out‚34498.01953125# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17) disk./.usagePercent‚75.79094222577693 disk.nvme1n1.in‚ 229.87109375# disk.nvme1n1.out‚34498.01953125 disk.md0.out‚ 605.11328125# disk.dm-0.out‚97036.33740234375# disk.nvme0c0n1.out‚ 78258.859375 disk.nvme0n1.in‚ 295.8828125! disk.nvme0n1.out‚ 78258.859375$ memory_percent‚97.00112342834473' proc.memory.availableMB‚ 491.3359375) proc.memory.percent‚18.46158504486084 network.recv‚ 3.078922e+06% disk./.usageGB‚1163.5804405212402 disk.dm-0.in‚ 525.75390625 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 295.8828125 cpu‚25.691590738497382 network.sent‚ 2.0765205e+07¼;ŒtžÖ)‚(:“ º¯öÒ¨×œÌ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚276.673' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.22433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ü3Ó4×)/ _wandbruntime‚11761  _runtime‚11761ó0¿ÌØ)Æ _step‚1064 ' lr-AdamW/pg1‚0.00017620351832559267 ' lr-AdamW/pg2‚0.00017620351832559267 trainer/global_step‚12949 % _timestamp‚1.7845185876303573e+09 _runtime‚11761.206547131ä¥ }ÑÙ)Ë trainer/global_step‚12949 % _timestamp‚1.7845185876303573e+09 _runtime‚11761.206547131 _step‚1064 ' lr-AdamW/pg1‚0.00017620351832559267 ' lr-AdamW/pg2‚0.00017620351832559267¨NV`4Ú)/ _wandbruntime‚11761  _runtime‚11761ø³½ (Û)†P VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1997389048102400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128140123 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53507999 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845185876310377e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104853111 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1985763496968645 _step‚1065 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1970742899160144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53508284 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104853109 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430535263317504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22592625 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430535244705280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15020614847861 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15043307807744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1997388689638912 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1997389048102400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104853111 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1985763854569279 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15128383675392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15020955409135 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104853109 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1982345398870528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15043307807744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232993234 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128129980 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128129980 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1997388689638912¿½ VÜ)‹P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104853111 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1985763496968645 _step‚1065 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1970742899160144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53508284 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104853109 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430535263317504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22592625 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430535244705280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15020614847861 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15043307807744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1997388689638912 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1997389048102400 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104853111 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1985763854569279 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15128383675392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15020955409135 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104853109 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1982345398870528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15043307807744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232993234 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128129980 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128129980 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1997388689638912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1997389048102400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128140123 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53507999 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845185876310377e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53512039 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1970890329827616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30917680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1985912393625115 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1997538432710656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233010697 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128149793 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15044417906688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1982493673372672 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1982493656333312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚1066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104860902 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1997538432710656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430568424975360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15044417906688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22594362 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233010697 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128139636 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚12949 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104860904 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233000538 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1985912036024369 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15021723236113 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22594359 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15044759337984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30917970 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚11761.821474442 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233000538 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1997538074240000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128149793 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚445698047939584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚445697917773824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15044417906688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22594362 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233010697 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128139636 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚12949 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104860904 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233000538 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1985912036024369 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15021723236113 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22594359 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15044759337984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30917970 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚11761.821474442 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233000538 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1997538074240000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128149793 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚445698047939584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚445697917773824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53512039 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1970890329827616 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30917680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1985912393625115 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1997538432710656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233010697 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128149793 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15044417906688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647ª®.ö4à)/ _wandbruntime‚11761  _runtime‚11761®á)‰ 0loss/FRt2/assignment_loss‚0.7966450256656628 2loss/SRqqt1/assignment_loss‚0.4306031334752064 2loss/SRqqt2/assignment_loss‚0.5842005674611466 3loss/FBt1/assignment_loss‚0.0043862481620260685 1loss/SRqqt1/detection_loss‚0.6084317155455581 1loss/SRqqt2/detection_loss‚0.6752806071717213 trainer/global_step‚12949 epoch‚32 3loss/FBt2/assignment_loss‚0.0008045355219278368 /loss/FRt1/detection_loss‚0.4612984692957808 /loss/FRt2/detection_loss‚1.7038557617847894 0loss/FBt1/detection_loss‚0.32909368481800083 .loss/FBt2/detection_loss‚0.694583137826218 %loss/total_loss‚7.853693545958064 % _timestamp‚1.7845185882467012e+09 _runtime‚11761.821955832 0loss/FRt1/assignment_loss‚1.5645106592300264 _step‚1067ºœ¼ââ)Ž _step‚1067 0loss/FRt2/assignment_loss‚0.7966450256656628 2loss/SRqqt1/assignment_loss‚0.4306031334752064 2loss/SRqqt2/assignment_loss‚0.5842005674611466 3loss/FBt1/assignment_loss‚0.0043862481620260685 1loss/SRqqt1/detection_loss‚0.6084317155455581 1loss/SRqqt2/detection_loss‚0.6752806071717213 trainer/global_step‚12949 epoch‚32 3loss/FBt2/assignment_loss‚0.0008045355219278368 /loss/FRt1/detection_loss‚0.4612984692957808 /loss/FRt2/detection_loss‚1.7038557617847894 0loss/FBt1/detection_loss‚0.32909368481800083 .loss/FBt2/detection_loss‚0.694583137826218 %loss/total_loss‚7.853693545958064 % _timestamp‚1.7845185882467012e+09 _runtime‚11761.821955832 0loss/FRt1/assignment_loss‚1.5645106592300264«–>zã)‚(:ö ɯöÒˆ¢ÕË" disk.nvme1c1n1.in‚ 229.87109375) proc.memory.percent‚18.46158504486084 cpu‚25.6913258277817 proc.cpu.threads‚17 network.recv‚ 3.081983e+06) disk./.usagePercent‚75.79300456007458 disk.md0.in‚0# disk.dm-1.out‚14461.56396484375! disk.nvme0c0n1.in‚ 295.8828125 disk.nvme0n1.in‚ 295.8828125' proc.memory.availableMB‚ 491.0546875 network.sent‚ 2.0768169e+07$ disk./.usageGB‚1163.612102508545& disk.nvme1n1.out‚34498.48388671875 disk.dm-0.in‚ 525.75390625# disk.dm-0.out‚97039.77978515625 disk.rbd0.out‚0( disk.nvme1c1n1.out‚34498.48388671875$ memory_percent‚97.00284004211426 disk.md0.out‚ 605.41015625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚78262.58251953125# proc.memory.rssMB‚ 3024.74609375 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0& disk.nvme0n1.out‚78262.58251953125R-ƒÉžä)‚(:“ ɯöÒ×Å€Ì gpu.0.gpu‚56.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚275.936' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.97866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í´vQvå)‚(:ë دöÒ“ÙË disk.md0.in‚0 disk.md0.out‚ 605.5859375 network.sent‚ 2.0832548e+07 network.recv‚ 3.086122e+06! disk.nvme0c0n1.in‚ 295.8828125# disk.nvme0c0n1.out‚ 78326.015625 disk.nvme0n1.in‚ 295.8828125& disk.nvme1n1.out‚34498.86669921875" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-0.in‚ 525.75390625# disk.dm-0.out‚97103.04931640625 disk.dm-1.in‚0$ memory_percent‚97.00133800506592( proc.memory.availableMB‚ 491.30078125# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084) disk./.usagePercent‚75.80046325295974% disk./.usageGB‚1163.7266120910645 disk.rbd0.in‚0 proc.cpu.threads‚17! disk.nvme0n1.out‚ 78326.015625( disk.nvme1c1n1.out‚34498.86669921875 disk.nvme1n1.in‚ 229.87109375! disk.dm-1.out‚14461.705078125 disk.rbd0.out‚0 cpu‚25.6910614258809›D³žæ)‚(:“ دöÒüú˜Ì gpu.0.gpu‚68.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚121.619' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.53966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vÿ¾4ç)/ _wandbruntime‚11803  _runtime‚11803õ>þÌè)Æ % _timestamp‚1.7845186301495311e+09 _runtime‚11803.725129138 _step‚1068 ' lr-AdamW/pg1‚0.00017398059849880657 ' lr-AdamW/pg2‚0.00017398059849880657 trainer/global_step‚12999*>Z_Ñé)Ë ' lr-AdamW/pg1‚0.00017398059849880657 ' lr-AdamW/pg2‚0.00017398059849880657 trainer/global_step‚12999 % _timestamp‚1.7845186301495311e+09 _runtime‚11803.725129138 _step‚1068¬öMw 4ê)/ _wandbruntime‚11803  _runtime‚11803ôMugáë)†P G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31030218 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1993190513555653 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15076098061269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15098876909568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105242761 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2004857983847936 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1989759106938368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53709838 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚447378465505280 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15184308370944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22679325 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚11803.729007783 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚432194276821504 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233857391 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128614632 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53709543 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚4473785851924^ææF+48 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128614632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15099218333696 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1993190871156287 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2004858342311424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 % _timestamp‚1.7845186301504672e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128624775 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31030510 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1978114415494384 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105242759 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1978114432533744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233867536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22679328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1989759106938368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚432194258209280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105242761 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128624775 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2004857983847936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105242759 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2004858342311424 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1989759123977728 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1989759123977728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0µï³m(ì)‹P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1989759123977728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31030218 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1993190513555653 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15076098061269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15098876909568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105242761 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2004857983847936 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1989759106938368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53709838 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚447378465505280 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15184308370944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22679325 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚11803.729007783 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚432194276821504 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233857391 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128614632 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53709543 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚447378585192448 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128614632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15099218333696 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1993190871156287 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2004858342311424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 % _timestamp‚1.7845186301504672e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128624775 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31030510 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1978114415494384 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105242759 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1978114432533744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233867536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22679328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1989759106938368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚432194258209280 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105242761 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128624775 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2004857983847936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105242759 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2004858342311424 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1989759123977728­7Žßá4í)/ _wandbruntime‚11804  _runtime‚11804Í'î)N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15077206449521 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1978261863201216 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1989907398479872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128624288 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53713871 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22681059 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31032809 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15100328439808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105250554 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1989907381440512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚1070 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128634445 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚447412760018432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15077547010907 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1993339052611377 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2005007368449024 trainer/global_step‚12999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15099987008512 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1989907381440512 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15100328439808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22681062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31032522 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1978261846161856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128624288 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53713581 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚447412885989888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15185428898304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚11804.338211539 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233874840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2005007726919680 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233884999 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374p{Ÿ Wï)•N OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2005007726919680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1989907381440512 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15100328439808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22681062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31032522 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1978261846161856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128624288 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53713581 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚447412885989888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15185428898304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚11804.338211539 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233874840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2005007726919680 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233884999 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15077206449521 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1978261863201216 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1989907398479872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128624288 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53713871 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22681059 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31032809 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15100328439808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105250554 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1989907381440512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚1070 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128634445 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚447412760018432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15077547010907 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 T?DeviceStatsMonitor.on_train_batch_end/requested_Æ®›éÄbytes.all.freed‚1993339052611377 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2005007368449024 trainer/global_step‚12999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15099987008512 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1997172849084928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31144967 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2012327278056960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22766025 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449093415364608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2012327636520448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53910713 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1985485948867984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15131581274677 _runtime‚11845.660736584 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1985485965907344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 234741838 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449093283094528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15131921835951 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 234731693 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129109427 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22766028 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15154787435520 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105632409 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15154446011392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31144688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2000617887743295 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105632411 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105632409 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15154787435520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚13049 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15240125039104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129109427 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1997172849084928 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1997172832045568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105632411 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1997172832045568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0X©›P(þ)ŠP QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129109427 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1997172849084928 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1997172832045568 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105632411 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1997172832045568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1997172849084928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31144967 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2012327278056960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22766025 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449093415364608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2012327636520448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53910713 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1985485948867984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15131581274677 _runtime‚11845.660736584 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1985485965907344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 234741838 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449093283094528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15131921835951 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 234731693 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129109427 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22766028 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15154787435520 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105632409 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15154446011392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31144688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2000617887743295 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105632411 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105632409 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15154787435520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚13049 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15240125039104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0±þX ‰4ÿ)/ _wandbruntime‚11846  _runtime‚118460lì¼€*N N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105640202 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1997321123587072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 234749142 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1997321106547712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105640204 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1985633379535456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2012476662658048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449127577861632 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15155556110336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1997321123587072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234759301 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1985633396574816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22767762 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚433886470595584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚433886451983360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.pea¬È¦Zk‚ 560516799 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105640202 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129119097 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2012477021128704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚11846.270276972 _step‚1074 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129108940 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15155897541632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15132689662929 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53914991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129108940 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1997321106547712 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15155556110336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15241235334656 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53914696 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2012477021128704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31147229 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚13049 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234759301 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22767759 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1997321123587072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234759301 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1985633396574816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22767762 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚433886470595584 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚433886451983360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105640202 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129119097 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2012477021128704 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚11846.270276972 _step‚1074 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129108940 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15155897541632 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15132689662929 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53914991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129108940 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1997321106547712 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15155556110336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15241235334656 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53914696 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2012477021128704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31147229 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚13049 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234759301 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22767759 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105640202 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1997321123587072²Íë4‚*/ _wandbruntime‚11846  _runtime‚11846j̃áƒ*ˆ _runtime‚11846.273659916 2loss/SRqqt1/assignment_loss‚0.4636475394590267 2loss/SRqqt2/assignment_loss‚0.5537180741686926 /loss/FRt1/detection_loss‚0.4942835101245998 trainer/global_step‚13049 _step‚1075 epoch‚32 4loss/FBt2/assignment_loss‚0.00029240711079510037 /loss/FBt2/detection_loss‚0.5029602185854487 1loss/SRqqt1/detection_loss‚0.6659742915295536 % _timestamp‚1.7845186726982932e+09 %loss/total_loss‚7.435433571264819 /loss/FRt1/assignment_loss‚1.487109037112539 /loss/FRt2/assignment_loss‚0.761915291657223 2loss/FBt1/assignment_loss‚0.005446471402024795 /loss/FRt2/detection_loss‚1.5515123147115333 1loss/SRqqt2/detection_loss‚0.6310673802066594 0loss/FBt1/detection_loss‚0.31750703519672324nº´Ø„* 2loss/FBt1/assignment_loss‚0.005446471402024795 /loss/FRt2/detection_loss‚1.5515123147115333 1loss/SRqqt2/detection_loss‚0.6310673802066594 0loss/FBt1/detection_loss‚0.31750703519672324 _runtime‚11846.273659916 2loss/SRqqt1/assignment_loss‚0.4636475394590267 2loss/SRqqt2/assignment_loss‚0.5537180741686926 /loss/FRt1/detection_loss‚0.4942835101245998 trainer/global_step‚13049 _step‚1075 epoch‚32 4loss/FBt2/assignment_loss‚0.00029240711079510037 /loss/FBt2/detection_loss‚0.5029602185854487 1loss/SRqqt1/detection_loss‚0.6659742915295536 % _timestamp‚1.7845186726982932e+09 %loss/total_loss‚7.435433571264819 /loss/FRt1/assignment_loss‚1.487109037112539 /loss/FRt2/assignment_loss‚0.761915291657223³Jô}…*‚(:ò ”°öÒóÐãË" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚36416.10888671875! disk.dm-0.out‚99126.455078125 cpu‚25.6900739070075( disk.nvme0c0n1.out‚78435.93798828125& disk.nvme0n1.out‚78435.93798828125 disk.md0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚14462.1494140625) proc.memory.percent‚18.46158504486084 network.sent‚ 2.0920638e+07 network.recv‚ 3.100323e+06 disk.nvme0n1.in‚ 295.8828125 disk.dm-0.in‚ 525.75390625# memory_percent‚97.0008134841919# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17% disk./.usageGB‚1166.4889945983887 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 295.8828125( disk.nvme1c1n1.out‚36416.10888671875 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 607.1171875( proc.memory.availableMB‚ 491.38671875( disk./.usagePercent‚75.9803937207874 disk.rbd0.out‚0@”e·ž†*‚(:“ ”°öÒ‘ñ¨Ì gpu.0.gpu‚61.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚292.733' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.57766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|ï™.ž‡*‚(:“ £°öÒÓÇË gpu.0.gpu‚92.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚257.308' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.76933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«ÙÌ~ˆ*‚(:ó £°öÒº¢›Ì cpu‚25.68981092722383! disk.nvme0c0n1.in‚ 295.8828125" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 607.55859375( proc.memory.availableMB‚ 491.13671875# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17% disk./.usageGB‚1166.4430084228516" disk.dm-1.out‚14462.2509765625 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚79086.6572265625 disk.nvme0n1.in‚ 295.8828125' disk.nvme1c1n1.out‚36637.9365234375$ memory_percent‚97.00233936309814 network.recv‚ 3.104401e+06) disk./.usagePercent‚75.97739836657557% disk.nvme0n1.out‚79086.6572265625 disk.md0.in‚0" disk.dm-0.out‚99997.9560546875) proc.memory.percent‚18.46158504486084 network.sent‚ 2.0972216e+07 disk.dm-1.in‚0 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚36637.9365234375 disk.dm-0.in‚ 525.75390625XÉiPy‰*‚(:î ²°öÒܤ§Ë disk.md0.out‚ 608.0859375! disk.dm-0.out‚99999.736328125# disk.dm-1.out‚14462.34912109375" disk.nvme1c1n1.in‚ 229.87109375# memory_percent‚97.0008134841919 proc.cpu.threads‚17 network.recv‚ 3.108059e+06% disk./.usageGB‚1166.4468727111816 disk.dm-0.in‚ 525.75390625! disk.nvme0c0n1.in‚ 295.8828125$ disk.nvme1c1n1.out‚ 36638.6484375 disk.nvme1n1.in‚ 229.87109375" disk.nvme1n1.out‚ 36638.6484375( proc.memory.availableMB‚ 491.38671875 disk.dm-1.in‚0( disk.nvme0c0n1.out‚79088.94189453125 disk.nvme0n1.in‚ 295.8828125& disk.nvme0n1.out‚79088.94189453125# proc.memory.rssMB‚ 3024.74609375 network.sent‚ 2.0977059e+07) disk./.usagePercent‚75.97765007074949 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚18.46158504486084 cpu‚25.689512650538145 disk.md0.in‚0~gNJžŠ*‚(:“ ²°öÒÕïË gpu.0.gpu‚39.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚291.079' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.02633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08h÷T4‹*/ _wandbruntime‚11888  _runtime‚11888î(èÌŒ*Æ ' lr-AdamW/pg1‚0.00016956057898422623 ' lr-AdamW/pg2‚0.00016956057898422623 trainer/global_step‚13099 % _timestamp‚1.7845187145018628e+09 _runtime‚11888.077483455 _step‚1076ÚÆ1åÑ*Ë % _timestamp‚1.7845187145018628e+09 _runtime‚11888.077483455 _step‚1076 ' lr-AdamW/pg1‚0.00016956057898422623 ' lr-AdamW/pg2‚0.00016956057898422623 trainer/global_step‚13099´eYÐ4Ž*/ _wandbruntime‚11888  _runtime‚11888ÄLD (*„P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 $ _timestamp‚1.784518714502876e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2019796930729472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1992857499280944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2004586557152768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2004586557152768 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15210015113216 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235616140 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106022059 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22852725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129594079 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2019796572265984 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15295816612864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129583936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚11888.078402578 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106022061 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2004586574192128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2008044904330303 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2019796572265984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106022059 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚13099 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15210015113216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15295913493504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15187405049359 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235605995 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15210356537344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22852728 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2004586574192128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106022061 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1992857482241584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15187064488085 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚450808101830144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1äk»b*‰P JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2019796572265984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106022059 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚13099 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15210015113216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15295913493504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15187405049359 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235605995 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15210356537344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22852728 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2004586574192128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106022061 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1992857482241584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15187064488085 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚450808101830144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 $ _timestamp‚1.784518714502876e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2019796930729472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1992857499280944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2004586557152768 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2004586557152768 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15210015113216 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235616140 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106022059 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22852725 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129594079 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2019796572265984 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15295816612864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129583936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚11888.078402578 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106022061 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2004586574192128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2008044904330303µy Mé4‘*/ _wandbruntime‚11888  _runtime‚11888Á0¢'’*N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2004734848694272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2008193443386139 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106029854 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235623444 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15296930274304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1993004929948416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2019945956867072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15188172876337 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2004734831654912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2019946315337728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31261767 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106029852 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2008193085785393 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129603749 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235633603 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2004734831654912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235633603 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235623444 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚450842395761664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚435545484099584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22854462 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54116226 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1078 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129593592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚13099 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚435545465487360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15297033439232 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1993004912909056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129603749 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106029852 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54116226 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1078 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129593592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚13099 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚435545465487360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15297033439232 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1993004912909056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129603749 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106029852 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2004734848694272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2008193443386139 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106029854 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235623444 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15296930274304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1993004929948416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2019945956867072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15188172876337 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2004734831654912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2019946315337728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31261767 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106029852 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2008193085785393 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129603749 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235633603 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2004734831654912 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235633603 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235623444 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚450842395761664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚435545484099584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22854462 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528¶$:j½4”*/ _wandbruntime‚11888  _runtime‚11888 úâ9•*Š 1loss/SRqqt1/detection_loss‚0.5812436353572754 1loss/SRqqt2/detection_loss‚0.6462170547689118 trainer/global_step‚13099 _runtime‚11888.693574133 _step‚1079 /loss/FBt1/detection_loss‚0.3131411782918988 0loss/FRt1/assignment_loss‚1.4965190138411886 3loss/SRqqt2/assignment_loss‚0.45059901673385866 2loss/FBt2/assignment_loss‚0.004346289101870837 /loss/FRt2/detection_loss‚1.5396292451959666 2loss/SRqqt1/assignment_loss‚0.4790750649739957 3loss/FBt1/assignment_loss‚0.0017748850976120752 epoch‚32 % _timestamp‚1.7845187151177719e+09 0loss/FBt2/detection_loss‚0.43653331523856675 %loss/total_loss‚7.140171421502968 0loss/FRt2/assignment_loss‚0.7379894009636491 /loss/FRt1/detection_loss‚0.4531033219381736É0ýí–* 0loss/FBt2/detection_loss‚0.43653331523856675 %loss/total_loss‚7.140171421502968 0loss/FRt2/assignment_loss‚0.7379894009636491 /loss/FRt1/detection_loss‚0.4531033219381736 1loss/SRqqt1/detection_loss‚0.5812436353572754 1loss/SRqqt2/detection_loss‚0.6462170547689118 trainer/global_step‚13099 _runtime‚11888.693574133 _step‚1079 /loss/FBt1/detection_loss‚0.3131411782918988 0loss/FRt1/assignment_loss‚1.4965190138411886 3loss/SRqqt2/assignment_loss‚0.45059901673385866 2loss/FBt2/assignment_loss‚0.004346289101870837 /loss/FRt2/detection_loss‚1.5396292451959666 2loss/SRqqt1/assignment_loss‚0.4790750649739957 3loss/FBt1/assignment_loss‚0.0017748850976120752 epoch‚32 % _timestamp‚1.7845187151177719e+09·_¯¬û]—*‚(:í Á°öÒÛ¤ŒÌ! disk.nvme0c0n1.in‚ 295.8828125 disk.nvme0n1.in‚ 295.8828125 disk.nvme1n1.in‚ 229.87109375" disk.nvme1n1.out‚ 36640.5234375" memory_percent‚97.001051902771) proc.memory.percent‚18.46158504486084 cpu‚25.689606245115556 network.sent‚ 2.0992866e+07 disk.dm-0.in‚ 525.75390625 disk.dm-1.in‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚79101.353515625" disk.nvme1c1n1.in‚ 229.87109375( proc.memory.availableMB‚ 491.34765625# proc.memory.rssMB‚ 3024.74609375 network.recv‚ 3.111164e+06 disk.md0.in‚0# disk.dm-1.out‚14462.41552734375 disk.rbd0.out‚0& disk.nvme0c0n1.out‚79101.353515625$ disk.nvme1c1n1.out‚ 36640.5234375 proc.cpu.threads‚17) disk./.usagePercent‚75.97867080198982 disk.md0.out‚ 608.41015625$ disk.dm-0.out‚100013.24560546875% disk./.uQ£¿ÁsageGB‚1166.46254348754886¦@ž˜*‚(:“ Á°öÒ¨¸ßÌ gpu.0.gpu‚77.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚287.171' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.72366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)}P9|™*‚(:ñ аöÒÔâÉË" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-1.in‚0# disk.dm-1.out‚14463.09423828125 disk.rbd0.out‚0 cpu‚25.689852252806663 network.sent‚ 2.1042518e+07" disk.nvme0c0n1.in‚ 295.91015625( proc.memory.availableMB‚ 491.07421875# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17 disk.md0.out‚ 609.61328125$ disk.dm-0.out‚100084.26318359375 network.recv‚ 3.115652e+06) disk./.usagePercent‚75.87320426837802% disk.nvme0n1.out‚79172.6884765625% disk.nvme1c1n1.out‚36643.34765625# disk.nvme1n1.out‚36643.34765625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 295.91015625 disk.nvme1n1.in‚ 229.87109375' disk.nvme0c0n1.out‚79172.6884765625# memory_percent‚97.0027208328247) proc.memory.percent‚18.46158504486084 disk.dm-0.in‚ 525.78125% disk./.usageGB‚1164.8433685302734^Fçåžš*‚(:“ аöÒ¼£ÕË gpu.0.gpu‚79.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚284.724' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.90799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸¥zþ{›*‚(:ð ß°öÒìà’Ë disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 296.37109375% disk.nvme0n1.out‚80579.8955078125" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 611.140625 cpu‚25.690306594916294 network.recv‚ 3.118882e+06) disk./.usagePercent‚75.89621619202961$ disk.dm-0.out‚101491.88427734375 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 296.37109375' disk.nvme0c0n1.out‚80579.8955078125$ disk.nvme1c1n1.out‚ 36646.9609375 disk.nvme1n1.in‚ 229.87109375$ memory_percent‚97.00119495391846# proc.memory.rssMB‚ 3024.74609375 network.sent‚ 2.1047397e+07% disk./.usageGB‚1165.1966590881348 disk.dm-1.in‚0" disk.nvme1n1.out‚ 36646.9609375 disk.md0.in‚0 disk.dm-0.in‚ 526.2421875( proc.memory.availableMB‚ 491.32421875) proc.memory.percent‚18.46158504486084# disk.dm-1.out‚14463.16064453125 þØœ*‚(:’ ß°öÒˆ±äË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.129' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.376333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nªœª4*/ _wandbruntime‚11931  _runtime‚11931Ìh¥Ìž*Æ _step‚1080 ' lr-AdamW/pg1‚0.00016736376318602552 ' lr-AdamW/pg2‚0.00016736376318602552 trainer/global_step‚13149 % _timestamp‚1.7845187575239232e+09 _runtime‚11931.099499484é 8ÉÑŸ*Ë % _timestamp‚1.7845187575239232e+09 _runtime‚11931.099499484 _step‚1080 ' lr-AdamW/pg1‚0.00016736376318602552 ' lr-AdamW/pg2‚0.00016736376318602552 trainer/global_step‚13149¸Ãå'4 */ _wandbruntime‚11931  _runtime‚11931‰.Š[ (¡*…P VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2027265866475008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106411709 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15242888262767 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2027266224938496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2012000282259968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15265925639168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54313825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22939425 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437171317333504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31374400 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2015471920917311 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15351741892608 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130078731 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15265584215040 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31374685 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2012000299299328 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15265925639168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236490442 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15265584215040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22939428 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437171298721280 _step‚1081 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2027265866475008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130078731 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2012000299299328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15265584215040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22939428 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437171298721280 _step‚1081 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2027265866475008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130078731 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2012000299299328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2027266224938496 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2012000282259968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15265925639168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54313825 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22939425¹7×¢4£*/ _wandbruntime‚11931  _runtime‚11931+ïK_!¤*N VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2012148573801472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15352765165568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106419502 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15243656089745 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22941162 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437204478991360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2000376446282656 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚11931.713887063 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15267035745280 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2027415609546752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2012148573801472 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130078244 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54318110 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚13149 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15266694313984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130088401 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2027415251076096 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106419504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452557244156928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236507905 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2012148556762112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437204497603584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2015620102372401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31376686 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7845187581381352e+09 OADeviceStatsMonitor.on_trainÁ}éo¶_batch_end/active.large_pool.allocated‚ 106419504 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106419502 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2000376463322016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15267035745280 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2027415609546752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236507905 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54317845 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130078244 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184‰I'¥*”N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚452557357545472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236497746 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15352859941888 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130088401 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2027415251076096 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106419504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452557244156928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236507905 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2012148556762112 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437204497603584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2015620102372401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31376686 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7845187581381352e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106419504 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106419502 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2000376463322016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15267035745280 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2027415609546752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236507905 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54317845 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130078244 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2012148573801472 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15352765165568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106419502 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15243656089745 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22941162 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437204478991360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2000376446282656 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚11931.713887063 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15267035745280 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2027415609546752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2012148573801472 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130078244 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54318110 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚13149 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15266694313984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2022898937504319 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2019414024406528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15298371476175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845187995192764e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2022898579903685 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚295 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106801359 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2019414024406528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15321494740992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23026128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _runtime‚11973.098063394 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31488230 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚454237785102336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2034735519147520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 237364744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106801361 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15407472877056 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2034735519147520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23026125 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15298030914901 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 237354599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2034735160684032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2034735160684032 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15321494740992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15321153316864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54514650 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚438830312225280 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130563383 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130553240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15321153316864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2007600566028144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2034735519147520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23026125 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15298030914901 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 237354599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VADevic :32eStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2034735160684032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2034735160684032 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15321494740992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15321153316864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54514650 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚438830312225280 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130563383 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130553240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15321153316864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2007600566028144 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2022898937504319 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2019414024406528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15298371476175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845187995192764e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2022898579903685½01T4µ*/ _wandbruntime‚11973  _runtime‚11973™µÅ'¶*N K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54518310 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106809152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237382207 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54518597 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31490735 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438863492495360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2019562298908672 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2034884545285120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚438863511107584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15322604847104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 237372048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15408595422208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15299479864539 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2007747996695616 _runtime‚11973.706078678 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚13199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130562896 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2034884903755776 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2034884903755776 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2007747979656256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15322263415808 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106809154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23027862 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚454272207597568 % _timestamp‚1.7845188001303678e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15322604847104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2019562281869312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚454272087917568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2023047118959409 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54518310 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106809152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237382207 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54518597 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31490735 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438863492495360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2019562298908672 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2034884545285120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚438863511107584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15322604847104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 237372048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15408595422208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15299479864539 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2007747996695616 _runtime‚11973.706078678 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚13199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130562896 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2034884903755776 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2034884903755776 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2007747979656256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15322263415808 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106809154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23027862 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚454272207597568 % _timestamp‚1.7845188001303678e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15322604847104 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2019562281869312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚454272087917568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2023047118959409 F&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 jet/accuracy_2_of_5‚1 jet/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981237053871155 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2735671103000641 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3046397864818573 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6143516898155212 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 'particle/accuracy‚0.856902539730072 _step‚1088 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6136025786399841 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26365047693252563 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7690212726593018 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5978283882141113 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6865109205245972 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.917907178401947 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3047853708267212 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_2_of_5‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.747381865978241 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1822747439146042 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20197352766990662 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6837341785430908 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24107596278190613 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6742886900901794 *jet/accuracy_2_of_3‚0.5575664639472961 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt2FBti¼U´º,/FRt_purity‚0.8781177997589111 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.29583555459976196 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3159905970096588 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16844196617603302 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11835252493619919 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6834447979927063 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6766048073768616 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7318906188011169 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7174323201179504 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6293788552284241 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 $ _timestamp‚1.784518815910486e+09 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.049437038600444794 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 *jet/accuracy_2_of_4‚0.7401526570320129 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.484375 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.954440176486969 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26973268389701843 +jet/accuracy_3_of_4‚0.09775324165821075 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3158891499042511 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09886959940195084 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04432476684451103 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5912942290306091 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3554142415523529 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2724454998970032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987531304359436 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6250340342521667 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8791193962097168 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.030199658125638962 )loss/val_total_loss‚7.501176103102595 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6259856224060059 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23091953992843628 particle/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6136025786399841 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9151235222816467 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.640841007232666 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5874999761581421 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 trainer/global_step‚13199 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 epoch‚32 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24699990451335907 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_4‚0.9803425073623657 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6908462643623352 particle/accuracy_3_of_3‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 *jet/accuracy_1_of_2‚0.7793475389480591 7validation_average_jet_accuracy‚0.38546130061149597 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968323707580566 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2624530494213104 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8129010200500488 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6577470302581787 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6899182796478271 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3166005611419678 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6397558450698853 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8791193962097168 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.045149363577365875 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8015684485435486 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_1_of_1‚1 +particle/specificity‚0.9133623242378235 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.529967725276947 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46528929471969604 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.617276132106781 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0610850490629673 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7274929285049438 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3220333755016327 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7365626096725464 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05106739327311516 'particle/f_score‚0.6612845063209534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9956710338592529 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8886620402336121 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4705359637737274 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6122105121612549 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6570631861686707 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9914901256561279 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.030199658125638962 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9962973594665527 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4516483545303345 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07847398519515991 *particle/sensitivity‚0.650471568107605 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6909782290458679 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7119428515434265 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6865385174751282 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.056742895394563675 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28275683522224426 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2607257068157196 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8129010200500488 particle/accuracy_5_of_5‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.044678278267383575 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 /particle/accuracy_4_of_4‚0.9552639722824097 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6506377458572388 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3047853708267212 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18219558894634247 particle/accuracy_2_of_4‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9913399815559387 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974259734153748 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.050027500838041306 _runtime‚11989.486503542 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3717758357524872 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36665067076683044 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6724371910095215 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3197557330131531 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19637632369995117 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4862295687198639 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.359474778175354 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8796305060386658 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7486186027526855 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6562594771385193 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.20000000298023224 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.7265625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8766251802444458 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt2SRqqt*FBt/event_purity‚0ömn=¿*èz =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6570631861686707 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9914901256561279 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.030199658125638962 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9962973594665527 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4516483545303345 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07847398519515991 *particle/sensitivity‚0.650471568107605 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6909782290458679 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7119428515434265 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6865385174751282 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.056742895394563675 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28275683522224426 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2607257068157196 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8129010200500488 particle/accuracy_5_of_5‚0 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.044678278267383575 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 /particle/accuracy_4_of_4‚0.9552639722824097 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6506377458572388 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3047853708267212 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18219558894634247 particle/accuracy_2_of_4‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9913399815559387 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974259734153748 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.050027500838041306 _runtime‚11989.486503542 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3717758357524872 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36665067076683044 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6724371910095215 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3197557330131531 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19637632369995117 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4862295687198639 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.359474778175354 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8796305060386658 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7486186027526855 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6562594771385193 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.20000000298023224 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.7265625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8766251802444458 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.3017868101596832 *jet/accuracy_2_of_2‚0.1628272533416748 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941260814666748 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.46487802267074585 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7132521867752075 particle/accuracy_1_of_4‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6544731855392456 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 +jet/accuracy_1_of_1‚0.34816431999206543 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20574985444545746 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31747299432754517 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6846898198127747 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.01670061983168125 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.891886293888092 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6905173063278198 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5879461765289307 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 jet/accuracy_2_of_5‚1 jet/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981237053871155 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2735671103000641 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.3046397864818573 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6143516898155212 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 'particle/accuracy‚0.856902539730072 _step‚1088 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6136025786399841 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26365047693252563 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7690212726593018 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5978283882141113 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6865109205245972 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.917907178401947 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3047853708267212 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_2_of_5‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.747381865978241 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1822747439146042 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20197352766990662 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6837341785430908 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.24107596278190613 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6742886900901794 *jet/accuracy_2_of_3‚0.5575664639472961 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8781177997589111 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.29583555459976196 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3159905970096588 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16844196617603302 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11835252493619919 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6834447979927063 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6766048073768616 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7318906188011169 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7174323201179504 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6293788552284241 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 $ _timestamp‚1.784518815910486e+09 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.049437038600444794 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 *jet/accuracy_2_of_4‚0.7401526570320129 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.484375 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.954440176486969 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26973268389701843 +jet/accuracy_3_of_4‚0.09775324165821075 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3158891499042511 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09886959940195084 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04432476684451103 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5912942290306091 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3554142415523529 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2724454998970032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987531304359436 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6250340342521667 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8791193962097168 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.030199658125638962 )loss/val_total_loss‚7.501176103102595 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6259856224060059 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.23091953992843628 particle/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6136025786399841 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9151235222816467 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.640841007232666 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5874999761581421 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 trainer/global_step‚13199 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 epoch‚32 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24699990451335907 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_4‚0.9803425073623657 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6908462643623352 particle/accuracy_3_of_3‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 *jet/accuracy_1_of_2‚0.7793475389480591 7validation_average_jet_accuracy‚0.38546130061149597 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968323707580566 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2624530494213104 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8129010200500488 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6577470302581787 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6899182796478271 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3166005611419678 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6397558450698853 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8791193962097168 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.045149363577365875 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8015684485435486 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_1_of_1‚1 +particle/specificity‚0.9133623242378235 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.529967725276947 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46528929471969604 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.617276132106781 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0610850490629673 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7274929285049438 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3220333755016327 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7365626096725464 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05106739327311516 'particle/f_score‚0.6612845063209534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9956710338592529 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8886620402336121 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4705359637737274 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6122105121612549À+15yÀ*‚(:î ª±öÒ¤éÚË# proc.memory.rssMB‚ 3024.74609375 cpu‚25.693006830428512 network.sent‚ 2.1199039e+07 network.recv‚ 3.136542e+06 disk.md0.in‚0 disk.dm-0.in‚ 530.234375 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚83149.947265625% disk.nvme1c1n1.out‚36946.92578125) disk./.usagePercent‚76.07309962305101$ disk.dm-0.out‚104357.00244140625 disk.nvme0n1.in‚ 300.36328125 disk.nvme1n1.in‚ 229.87109375$ memory_percent‚97.00374603271484) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17& disk.nvme0c0n1.out‚83149.947265625 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 300.36328125" disk.nvme1c1n1.in‚ 229.87109375% proc.memory.availableMB‚ 490.90625% disk./.usageGB‚1167.9122619628906 disk.md0.out‚ 613.18359375# disk.dm-1.out‚14463.67626953125# disk.nvme1n1.out‚36946.92578125(ª^RžÁ*‚(:“ ª±öÒÓ Ì gpu.0.gpu‚84.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚122.671' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.89033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ر‹-~Â*‚(:ó ¹±öÒƒ³àË$ memory_percent‚97.00376987457275) proc.memory.percent‚18.46158504486084 network.recv‚ 3.140869e+06 disk.nvme0n1.in‚ 300.36328125' disk.nvme1c1n1.out‚36947.5146484375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 613.4921875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 300.36328125 cpu‚25.6926828840494) disk./.usagePercent‚76.07334784858877% disk.nvme0n1.out‚83158.4814453125$ disk.dm-0.out‚104365.32763671875' disk.nvme0c0n1.out‚83158.4814453125# proc.memory.rssMB‚ 3024.74609375 disk.dm-0.in‚ 530.234375$ disk./.usageGB‚1167.916072845459" disk.nvme1c1n1.in‚ 229.87109375% disk.nvme1n1.out‚36947.5146484375( proc.memory.availableMB‚ 490.90234375 proc.cpu.threads‚17 network.sent‚ 2.1245014e+07 disk.md0.in‚0 disk.dm-1.in‚0# disk.dm-1.out‚14463.79833984375 disk.rbd0.in‚0î"! Ã*‚(:’ ¹±öÒ‡Ž¬Ì gpu.0.gpu‚96.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.36' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.45333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V½Í€Ä*‚(:õ ȱöÒìÞË) disk./.usagePercent‚76.07436261645282 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 613.82421875# disk.dm-1.out‚14463.84912109375% disk.nvme0n1.out‚83165.3427734375 network.sent‚ 2.12499e+07' disk.nvme1c1n1.out‚36947.9541015625 disk.dm-0.in‚ 530.234375 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 300.36328125 cpu‚25.692401533326063 proc.cpu.threads‚17% disk./.usageGB‚1167.9316520690918" disk.nvme1c1n1.in‚ 229.87109375% disk.nvme1n1.out‚36947.9541015625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚83165.3427734375 disk.nvme0n1.in‚ 300.36328125$ memory_percent‚97.00376987457275( proc.memory.availableMB‚ 490.90234375 network.recv‚ 3.144384e+06 disk.md0.in‚0$ disk.dm-0.out‚104371.87841796875 disk.rbd0.in‚0# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084iW‹Ç“Å*‚(:ˆ ȱöÒòü Ì gpu.0.gpu‚92.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚280.134' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.378 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë©xT4Æ*/ _wandbruntime‚12031  _runtime‚12031žLþßÉÇ*à $ _timestamp‚1.784518857586294e+09 _runtime‚12031.163019258 _step‚1089 & lr-AdamW/pg1‚0.0001629972249856097 & lr-AdamW/pg2‚0.0001629972249856097 trainer/global_step‚13249ÙWä1ÎÈ*È & lr-AdamW/pg2‚0.0001629972249856097 trainer/global_step‚13249 $ _timestamp‚1.784518857586294e+09 _runtime‚12031.163019258 _step‚1089 & lr-AdamW/pg1‚0.0001629972249856097Á,ú¨4É*/ _wandbruntime‚12031  _runtime‚12031 ´cR9Ê*†P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15374158538480 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚440511904391680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131129529 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2032267513948448 TFDeviceStatsMonitor.on_train_batch_start/reservedÏ”{Ó#_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 % _timestamp‚1.7845188575869284e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15484048386048 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2028770076349952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23116944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2044167475067904 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107313819 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31638101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15397040254464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚1090 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131119386 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107313817 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2044167475067904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54754763 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15397381678592 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2028770093389312 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54755045 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23116941 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15373817977206 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131129529 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2044167116604416 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2028770076349952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2032267156347814 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2016893355409968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2044167116604416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455995952777728 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31637822 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238443348 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚13249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107313817 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15397040254464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2016893338370608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚455996076659200õù:É(Ë*‹P ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15397040254464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚1090 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131119386 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107313817 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2044167475067904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54754763 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15397381678592 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2028770093389312 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54755045 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23116941 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15373817977206 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131129529 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2044167116604416 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2028770076349952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2032267156347814 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2016893355409968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2044167116604416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455995952777728 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31637822 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238443348 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚13249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107313817 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15397040254464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2016893338370608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚455996076659200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15374158538480 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚440511904391680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131129529 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2032267513948448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 % _timestamp‚1.7845188575869284e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15484048386048 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2028770076349952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23116944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2044167475067904 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107313819 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31638101 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÂqË6ü4Ì*/ _wandbruntime‚12031  _runtime‚120318¤»³'Í*N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131129042 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚440545084661760 trainer/global_step‚13249 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2044316859676160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15398150353408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2044316859676160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15485163317248 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107321610 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107321610 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15398491784704 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131139199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107321612 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2028918367891456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131129042 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107321612 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2028918350852096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238450652 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54759047 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1091 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238460811 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚440545103273984 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚456030373950464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15374926365458 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238460811 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131139199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2032416053004284 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54758762 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2017040769038080ý¨§° Î*”N \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚456030373950464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15374926365458 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238460811 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131139199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2032416053004284 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54758762 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2017040769038080 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131129042 VHDeviceStatsMonitor.on_train_batch_end°¸ð0j/requested_bytes.small_pool.current‚ 340561386 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚440545084661760 trainer/global_step‚13249 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2044316859676160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15398150353408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2044316859676160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15485163317248 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107321610 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107321610 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15398491784704 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131139199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107321612 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2028918367891456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131129042 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107321612 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2028918350852096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238450652 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54759047 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1091 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238460811 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚440545103273984 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232Ã…Ùy4Ï*/ _wandbruntime‚12031  _runtime‚12031xž\ Ð*… 2loss/FBt2/assignment_loss‚0.004017925426724631 0loss/FBt1/detection_loss‚0.34360819082257876 %loss/total_loss‚7.218330937930849 -loss/FRt2/detection_loss‚1.54116803743209 1loss/SRqqt1/detection_loss‚0.5538864453455611 1loss/SRqqt2/detection_loss‚0.7058218813949904 _step‚1092 3loss/SRqqt2/assignment_loss‚0.48642326783789114 .loss/FBt2/detection_loss‚0.459798030659279 % _timestamp‚1.7845188582011368e+09 epoch‚33 /loss/FRt1/assignment_loss‚1.457090672331326 0loss/FRt2/assignment_loss‚0.7610727620535883 2loss/SRqqt1/assignment_loss‚0.4252516250965597 2loss/FBt1/assignment_loss‚0.006187417051546518 /loss/FRt1/detection_loss‚0.4740046824787145 trainer/global_step‚13249 _runtime‚12031.776475391x”/gÑ*Š epoch‚33 /loss/FRt1/assignment_loss‚1.457090672331326 0loss/FRt2/assignment_loss‚0.7610727620535883 2loss/SRqqt1/assignment_loss‚0.4252516250965597 2loss/FBt1/assignment_loss‚0.006187417051546518 /loss/FRt1/detection_loss‚0.4740046824787145 trainer/global_step‚13249 _runtime‚12031.776475391 2loss/FBt2/assignment_loss‚0.004017925426724631 0loss/FBt1/detection_loss‚0.34360819082257876 %loss/total_loss‚7.218330937930849 -loss/FRt2/detection_loss‚1.54116803743209 1loss/SRqqt1/detection_loss‚0.5538864453455611 1loss/SRqqt2/detection_loss‚0.7058218813949904 _step‚1092 3loss/SRqqt2/assignment_loss‚0.48642326783789114 .loss/FBt2/detection_loss‚0.459798030659279 % _timestamp‚1.7845188582011368e+09ÄgÚ~Ò*‚(:ó ×±öÒåÉåÌ network.sent‚ 2.1252822e+07) disk./.usagePercent‚76.08044675110489" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375! disk.dm-0.out‚104422.11328125 disk.dm-1.in‚0 disk.dm-1.out‚14464.00390625 disk.rbd0.out‚0 network.recv‚ 3.147641e+06$ disk./.usageGB‚1168.025058746338& disk.nvme1n1.out‚36949.00634765625 disk.md0.in‚0 disk.dm-0.in‚ 530.234375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚83215.34228515625 disk.nvme0n1.in‚ 300.36328125 disk.md0.out‚ 614.12890625" disk.nvme0c0n1.in‚ 300.36328125# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 cpu‚25.69211629800354 proc.cpu.threads‚17( disk.nvme1c1n1.out‚36949.00634765625& disk.nvme0n1.out‚83215.34228515625$ memory_percent‚97.00415134429932( proc.memory.availableMB‚ 490.83984375Ç#CžÓ*‚(:“ ×±öÒ÷‡°Ì gpu.0.gpu‚97.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚267.038' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.01266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ¬9†uÔ*‚(:ê æ±öÒš½»Ë disk.dm-1.out‚ 14464.0703125& proc.memory.availableMB‚ 490.421875% disk./.usageGB‚1168.0317039489746 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-0.in‚ 530.234375 disk.dm-0.out‚104445.1640625 network.sent‚ 2.1317388e+07 network.recv‚ 3.152119e+06 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 300.36328125& disk.nvme0n1.out‚83238.66162109375 disk.md0.in‚0 disk.md0.out‚614.4375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚83238.66162109375# memory_percent‚97.0067024230957# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 cpu‚25.69182127073706 proc.cpu.threads‚17( disk./.usagePercent‚76.0808795928334 disk.nvme0n1.in‚ 300.36328125( disk.nvme1c1n1.out‚36949.48974609375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚36949.48974609375Ž‚œÕ*‚(:‘ æ±öÒøí†Ì gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.824' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.27466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ež‹s4Ö*/ _wandbruntime‚12073  _runtime‚12073k£€æË×*Å trainer/global_step‚13299 $ _timestamp‚1.784518899724572e+09 _runtime‚12073.302020491 _step‚1093 ' lr-AdamW/pg1‚0.00016082778303798335 ' lr-AdamW/pg2‚0.00016082778303798335Ýk…ÔÐØ*Ê ' lr-AdamW/pg1‚0.00016082778303798335 ' lr-AdamW/pg2‚0.00016082778303798335 trainer/global_step‚13299 $ _timestamp‚1.784518899724572e+09 _runtime‚12073.302020491 _step‚1093Å‚Íâ4Ù*/ _wandbruntime‚12073  _runtime‚120738-8m (Ú*†P QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15539982431744 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54956839 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2051636410813440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2051636769276928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239307505 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107703467 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2036183818496512 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2036183818496512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31753195 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2024264888783568 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131614181 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2024264871744208 % _timestamp‚1.7845188997252252e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107703467 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚442170936507904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15452950780416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23203644 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2051636769276928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2039694530535456 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131604038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23203641 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31752915 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2036183801457152 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131614181 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2039694172934822 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107703469 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54956556 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15429301190614 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457710790863872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚442170917895680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚13299 _step‚1094 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2051636410813440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457710918939648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚12073.302113241 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107703469 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15452609356288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2036183801457152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚442170936507904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15452950780416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23203644 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2051636769276928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2039694530535456 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131604038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23203641 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31752915 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2036183801457152 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131614181 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2039694172934822 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107703469 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54956556 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15429301190614 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457710790863872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚442170917895680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚13299 _step‚1094 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2051636410813440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457710918939648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚12073.302113241 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Æ$ïF,4Ü*/ _wandbruntime‚12074  _runtime‚12074^SvóVÝ*N =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15454060886528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15453719455232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 239324954 GADeviceStatsMonitor.on_train_batch_end/overu.ÏCÀ$size_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2024412319451040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54960799 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15430750140252 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131623851 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23205375 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚457745215993344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2024412302411680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚442204116777984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845189006449008e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2051786153885184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239335113 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚13299 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54960503 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239335113 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 239324954 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31755128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1095 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2051785795414528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15540981370368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15454060886528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15453719455232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚442204098165760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107711260 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107711260 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15454060886528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15453719455232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 239324954 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2024412319451040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54960799 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15430750140252 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131623851 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23205375 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚457745215993344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2024412302411680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚442204116777984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845189006449008e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2051786153885184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239335113 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚296 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚13299 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54960503 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239335113 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 239324954 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31755128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1095 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2051785795414528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15540981370368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15454060886528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15453719455232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚442204098165760 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224ÇÐýt©4ß*/ _wandbruntime‚12074  _runtime‚12074X'8 à*‡ 0loss/FRt1/assignment_loss‚1.5920908564944947 3loss/FBt1/assignment_loss‚0.0025501958373740784 1loss/SRqqt1/detection_loss‚0.6185179040340942 0loss/FRt2/assignment_loss‚0.7956072318240964 3loss/SRqqt1/assignment_loss‚0.41830069047549834 4loss/FBt2/assignment_loss‚0.00022528531619571575 -loss/FRt2/detection_loss‚1.51602613218551 1loss/SRqqt2/detection_loss‚0.6650237968427996 %loss/total_loss‚7.488980059605856 trainer/global_step‚13299 _runtime‚12074.22113964 epoch‚33 $ _timestamp‚1.784518900645862e+09 1loss/SRqqt2/assignment_loss‚0.500759648489779 /loss/FRt1/detection_loss‚0.5012474292158273 /loss/FBt1/detection_loss‚0.3829218024900544 0loss/FBt2/detection_loss‚0.49570908640013184 _step‚1096ÜŒá*Œ /loss/FRt1/detection_loss‚0.5012474292158273 /loss/FBt1/detection_loss‚0.3829218024900544 0loss/FBt2/detection_loss‚0.49570908640013184 _step‚1096 0loss/FRt1/assignment_loss‚1.5920908564944947 3loss/FBt1/assignment_loss‚0.0025501958373740784 1loss/SRqqt1/detection_loss‚0.6185179040340942 0loss/FRt2/assignment_loss‚0.7956072318240964 3loss/SRqqt1/assignment_loss‚0.41830069047549834 4loss/FBt2/assignment_loss‚0.00022528531619571575 -loss/FRt2/detection_loss‚1.51602613218551 1loss/SRqqt2/detection_loss‚0.6650237968427996 %loss/total_loss‚7.488980059605856 trainer/global_step‚13299 _runtime‚12074.22113964 epoch‚33 $ _timestamp‚1.784518900645862e+09 1loss/SRqqt2/assignment_loss‚0.500759648489779ÈδWÃ{â*‚(:ð õ±öÒˆ»©Í# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17' disk.nvme0c0n1.out‚83372.9755859375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚97.00841903686523 network.sent‚ 2.1333201e+07$ disk./.usageGB‚1168.031967163086" disk.nvme0c0n1.in‚ 300.36328125 disk.nvme0n1.in‚ 300.36328125% disk.nvme0n1.out‚83372.9755859375 disk.dm-0.out‚104579.3671875) proc.memory.percent‚18.46158504486084( disk./.usagePercent‚76.0808967375402 disk.md0.in‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 490.140625 cpu‚25.691595678332064 network.recv‚ 3.155371e+06( disk.nvme1c1n1.out‚36950.30322265625& disk.nvme1n1.out‚36950.30322265625 disk.md0.out‚ 614.7890625 disk.dm-0.in‚ 530.234375# disk.dm-1.out‚14464.23388671875wÊDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108093117 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15595682046976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55157546 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2043597543603712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15508519882240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2043597543603712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2043597526564352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23290344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23290341 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2047121189521830 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132098833 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443829931399680 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845189420051665e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31867202 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2059105705022464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KB„˜‘¦ DeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2031636422157168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚459425613446656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2031636405117808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55157257 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2043597526564352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2047121547122464 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132088690 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240191952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452G2XÐ(í*‹P LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2031636405117808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55157257 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2043597526564352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2047121547122464 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132088690 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240191952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15595791510528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15485124965296 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108093117 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15595682046976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55157546 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2043597543603712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15508519882240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2043597543603712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2043597526564352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23290344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23290341 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2047121189521830 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132098833 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443829931399680 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845189420051665e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31867202 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2059105705022464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2031636422157168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚459425613446656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0Êú†é4î*/ _wandbruntime‚12116  _runtime‚12116/hîñ'ï*N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23292075 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132108503 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2059255089623552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2043745801066496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108100912 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108100910 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55161340 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2043745818105856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55161626 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132108503 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15509288557056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15485892792274 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31869265 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15596797936640 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2031783852824640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15596907393024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15486233353660 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132098346 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108100912 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2059255448094208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2031783835785280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108100910 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚459459909606400 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240209415 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚12116.18978938 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2047270086178300 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528:S„³$ð*”N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2043745801066496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚459460037675008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚443863111669760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240209415 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚12116.18978938 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2047270086178300 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23292075 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132108503 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2059255089623552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2043745801066496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108100912 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108100910 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55161340 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2043745818105856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55161626 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132108503 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15509288557056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15485892792274 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31869265 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15596797936640 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2031783852824640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15596907393024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15486233353660 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132098346 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108100912 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2059255448094208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2031783835785280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108100910 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚459459909606400 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2054548206108838 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚461140570226176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2054548563709472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚13399 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2066575357694976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241056109 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461140450539008 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2066574999231488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2066575357694976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2051011268710912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23377041 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132573342 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7845189843306837e+09 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15540608178704 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2051011251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15563747559936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15651505635328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108482769 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108482767 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15651606710272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108482767 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15564088984064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132583485 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚12157.90799499 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚445488963515904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚1102 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2051011251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31982275 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2039007955530768 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132583485 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2051011268710912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184ú*7½(ÿ*ŠP aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2051011268710912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23377041 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132573342 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7845189843306837e+09 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15540608178704 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2051011251671552 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15563747559936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15651505635328 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108482769 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108482767 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15651606710272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108482767 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15564088984064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132583485 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚12157.90799499 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚445488963515904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚1102 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2051011251671552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31982275 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2039007955530768 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132583485 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2051011268710912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15563747559936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31981997 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108482769 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132573342 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241066254 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2054548206108838 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚461140570226176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2054548563709472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚13399 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2066575357694976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241056109 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461140450539008 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2066574999231488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2066575357694976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232ÎÖÛb+4€+/ _wandbruntime‚12158  _runtime‚12158 Q«¢é+N VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15564857658880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241083717 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23378778 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2039155369158880 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241073558 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31984511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 $ _timestamp‚1.784518984941356e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55363005 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132593155 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚12158.517078869 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚13399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132593155 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚445522143785984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15564857658880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2066724383832576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2051159526173696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461174742774784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132582998 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚445522125173760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108490560 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132582998 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31984230 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15652617601024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55363289 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241073558 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15541716567068 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108490562 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2066724742303232 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2051159543213056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2051159543213056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2[`¼þ'‚+”N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23378778 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2039155369158880 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241073558 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31984511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 $ _timestamp‚1.784518984941356e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55363005 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132593155 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚12158.517078869 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚13399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132593155 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚445522143785984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15564857658880 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2066724383832576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2051159526173696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461174742774784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132582998 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚445522125173760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108490560 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132582998 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31984230 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15652617601024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55363289 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241073558 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15541716567068 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108490562 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2066724742303232 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2051159543213056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2051159543213056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15564857658880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241083717 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452Ï"ÉP®4ƒ+/ _wandbruntime‚12158  _runtime‚12158¢%¦ó„+ˆ /loss/FRt1/detection_loss‚0.4851843931784414 /loss/FRt2/detection_loss‚1.4963176178663995 1loss/SRqqt1/detection_loss‚0.5827620831345246 trainer/global_step‚13399 % _timestamp‚1.7845189849439847e+09 3loss/SRqqt1/assignment_loss‚0.41175718203851713 2loss/SRqqt2/assignment_loss‚0.5043819856109284 0loss/FBt2/detection_loss‚0.30500534201133567 epoch‚33 _step‚1104 0loss/FRt1/assignment_loss‚1.5415184594647058 0loss/FRt2/assignment_loss‚0.7258522562300973 2loss/FBt1/assignment_loss‚0.009373032110121114 0loss/SRqqt2/detection_loss‚0.677795080978375 /loss/FBt1/detection_loss‚0.5081209185549856 %loss/total_loss‚7.271217262981469 _runtime‚12158.523006357 2loss/FBt2/assignment_loss‚0.023148911803036805âVN…+ trainer/global_step‚13399 % _timestamp‚1.7845189849439847e+09 3loss/SRqqt1/assignment_loss‚0.41175718203851713 2loss/SRqqt2/assignment_loss‚0.5043819856109284 0loss/FBt2/detection_loss‚0.30500534201133567 epoch‚33 _step‚1104 0loss/FRt1/assignment_loss‚1.5415184594647058 0loss/FRt2/assignment_loss‚0.7258522562300973 2loss/FBt1/assignment_loss‚0.009373032110121114 0loss/SRqqt2/detection_loss‚0.677795080978375 /loss/FBt1/detection_loss‚0.5081209185549856 %loss/total_loss‚7.271217262981469 _runtime‚12158.523006357 2loss/FBt2/assignment_loss‚0.023148911803036805 /loss/FRt1/detection_loss‚0.4851843931784414 /loss/FRt2/detection_loss‚1.4963176178663995 1loss/SRqqt1/detection_loss‚0.5827620831345246ÐcDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32096646 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 12200.2941849 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2046379488904368 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15619316661760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462855260547584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚13449 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133057994 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108872417 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133068137 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32096350 $ _timestamp‚1.784519026717831e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15596091392112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23463741 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚447147977019904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241940556 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15707411603456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15707302139904 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241930411 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55560390 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133057994 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15619658085888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133068137 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2061975580296480 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108872417 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2058424976778752 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15619316661760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55560091 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚447147958407680 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2061975222695846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23463744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462855388623360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚1106 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108872419 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2074044651904000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2046379471865008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2074044651904000 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2074044293440512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2074044293440512 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2058424993818112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3Ü$Ò®‹‘+ˆP F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241930411 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55560390 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133057994 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15619658085888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133068137 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2061975580296480 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108872417 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2058424976778752 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15619316661760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5556009‘=ù ƒ!1 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚447147958407680 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2061975222695846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23463744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462855388623360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _step‚1106 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108872419 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2074044651904000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2046379471865008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2074044651904000 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2074044293440512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2074044293440512 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2058424993818112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2058424976778752 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32096646 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚ 12200.2941849 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2046379488904368 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15619316661760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462855260547584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚13449 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133057994 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108872417 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133068137 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32096350 $ _timestamp‚1.784519026717831e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15596091392112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23463741 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚447147977019904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241940556 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15707411603456 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15707302139904Òï>î4’+/ _wandbruntime‚12200  _runtime‚12200,CKp'“+N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133077807 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2062123761751570 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2074193678041600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133067650 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2074194036512256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55564192 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2046526919571840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241947860 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2058573268320256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108880212 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241958019 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108880210 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15620426760704 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241958019 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23465478 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15596859219090 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2058573251280896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32098987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241947860 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15708528804864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15708417251328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚462889555929088 _step‚1107 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133077807 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108880210 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0ã3×g'”+•N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133067650 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2074194036512256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55564192 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2046526919571840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241947860 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2058573268320256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108880212 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241958019 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108880210 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15620426760704 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241958019 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23465478 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15596859219090 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2058573251280896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32098987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241947860 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15708528804864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15708417251328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚462889555929088 _step‚1107 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133077807 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108880210 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133077807 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2062123761751570 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2074193678041600 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3ÓÕîÅ4•+/ _wandbruntime‚12200  _runtime‚12200°Éõû –+‡ /loss/FRt2/detection_loss‚1.5402689130679352 0loss/FBt1/detection_loss‚0.42463797671448406 /loss/FBt2/detection_loss‚0.6308583443081288 %loss/total_loss‚7.637982337203707 % _timestamp‚1.7845190273323479e+09 3loss/FBt1/assignment_loss‚0.0004823484420184884 trainer/global_step‚13449 _step‚1108 0loss/FRt2/assignment_loss‚0.6860731894319708 2loss/SRqqt2/assignment_loss‚0.4754789275188064 1loss/FBt2/assignment_loss‚0.00381287152152768 /loss/FRt1/detection_loss‚0.4998265025036728 1loss/SRqqt1/detection_loss‚0.6158737966274396 1loss/SRqqt2/detection_loss‚0.6209151337166291 epoch‚33 _runtime‚12200.907726659 /loss/FRt1/assignment_loss‚1.637076280500588 2loss/SRqqt1/assignment_loss‚0.5026780528505058E1!Y—+Œ /loss/FRt1/assignment_loss‚1.637076280500588 2loss/SRqqt1/assignment_loss‚0.5026780528505058 /loss/FRt2/detection_loss‚1.5402689130679352 0loss/FBt1/detection_loss‚0.42463797671448406 /loss/FBt2/detection_loss‚0.6308583443081288 %loss/total_loss‚7.637982337203707 % _timestamp‚1.7845190273323479e+09 3loss/FBt1/assignment_loss‚0.0004823484420184884 trainer/global_step‚13449 _step‚1108 0loss/FRt2/assignment_loss‚0.6860731894319708 2loss/SRqqt2/assignment_loss‚0.4754789275188064 1loss/FBt2/assignment_loss‚0.00381287152152768 /loss/FRt1/detection_loss‚0.4998265025036728 1loss/SRqqt1/detection_loss‚0.6158737966274396 1loss/SRqqt2/detection_loss‚0.6209151337166291 epoch‚33 _runtime‚12200.907726659Ôßüu&|˜+‚(:ñ ü²öÒÎîÁË) proc.memory.percent‚18.46158504486084 cpu‚25.68912002228613) disk./.usagePercent‚76.09187556175004 disk.md0.out‚618.1875 disk.dm-0.in‚ 530.234375$ disk.dm-0.out‚104783.22509765625 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚83579.083984375 disk.nvme1n1.in‚ 229.87109375 network.sent‚ 2.1551235e+07 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 300.36328125& disk.nvme0c0n1.out‚83579.083984375 disk.nvme0n1.in‚ 300.36328125( disk.nvme1c1n1.out‚36956.24853515625# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17" disk.dm-1.out‚14465.0537109375& disk.nvme1n1.out‚36956.24853515625$ memory_percent‚97.00028896331787 network.recv‚ 3.188188e+06% disk./.usageGB‚1168.2005195617676 disk.md0.in‚0" disk.nvme1c1n1.in‚ 229.87109375( proc.memory.availableMB‚ 491.47265625LýLþž™+‚(:“ ü²öÒ­Œ‘Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚277.136' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.37866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûÑ^vpš+‚(:å ‹³öÒ ‚°Ë( disk./.usagePercent‚76.0918658712636" disk.nvme1c1n1.in‚ 229.87109375 disk.rbd0.out‚0% proc.memory.availableMB‚ 491.46875) proc.memory.percent‚18.46158504486084 disk.nvme1n1.in‚ 229.87109375 disk.nvme1n1.out‚ 36956.9375 disk.md0.out‚ 618.51171875 disk.dm-0.in‚ 530.234375 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚83581.498046875# proc.memory.rssMB‚ 3024.74609375 network.recv‚ 3.192082e+06$ disk.dm-0.out‚104785.52783203125# disk.dm-1.out‚14465.13134765625 disk.nvme0n1.in‚ 300.36328125 cpu‚25.688848054466213 proc.cpu.threads‚17 network.sent‚ 2.1601619e+07% disk./.usageGB‚1168.2003707885742! disk.nvme1c1n1.out‚ 36956.9375 disk.md0.in‚0" disk.nvme0c0n1.in‚ 300.36328125$ disk.nvme0n1.out‚83581.498046875$ memory_percent‚97.00031280517578/µ™Éî›+‚(:“ ‹³öÒݱ„Ì gpu.0.gpu‚84.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚219.795' gpu.0.enforcedPowerLimitWat6ê˶°ts‚300.0( gpu.0.powerPercent‚73.26499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–¨Prœ+‚(:ç š³öÒÓßÃË disk.dm-1.in‚0$ memory_percent‚97.00050354003906) proc.memory.percent‚18.46158504486084 cpu‚25.68857545001454 proc.cpu.threads‚17 network.sent‚ 2.1606548e+07) disk./.usagePercent‚76.09189419730093$ disk.nvme1n1.out‚36961.384765625 disk.dm-0.in‚ 530.234375# disk.dm-1.out‚14465.27880859375" disk.nvme0c0n1.in‚ 300.36328125# disk.nvme0c0n1.out‚ 83588.828125 disk.nvme0n1.in‚ 300.36328125 network.recv‚ 3.195598e+06% disk./.usageGB‚1168.2008056640625" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 621.3828125$ disk.dm-0.out‚104791.34716796875 disk.rbd0.out‚0# proc.memory.rssMB‚ 3024.74609375& disk.nvme1c1n1.out‚36961.384765625 disk.md0.in‚0 disk.rbd0.in‚0! disk.nvme0n1.out‚ 83588.828125$ proc.memory.availableMB‚491.4375z]#œ+‚(:‘ š³öÒ©Î„Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚121.508' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.50266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©I²S4ž+/ _wandbruntime‚12242  _runtime‚12242Z:¦8ÌŸ+Æ ' lr-AdamW/pg1‚0.00015224544038274125 ' lr-AdamW/pg2‚0.00015224544038274125 trainer/global_step‚13499 % _timestamp‚1.7845190689414303e+09 _runtime‚12242.519027178 _step‚1109ñ`QÑ +Ë trainer/global_step‚13499 % _timestamp‚1.7845190689414303e+09 _runtime‚12242.519027178 _step‚1109 ' lr-AdamW/pg1‚0.00015224544038274125 ' lr-AdamW/pg2‚0.00015224544038274125ÕCÇ‘4¡+/ _wandbruntime‚12242  _runtime‚12242„4 y (¢+…P [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _step‚1110 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133542646 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2081513587649536 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2065838718925312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32210807 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2053751022277968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109262069 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚448806971911680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15674885763584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133552789 _runtime‚12242.519089178 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15675227187712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15674885763584 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2069402596883488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2081513946113024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23550444 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚448806990523904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133542646 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32211096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2069402239282854 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2081513946113024 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2081513587649536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242814858 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚464570192914944 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 $ _timestamp‚1.784519068942044e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2065838701885952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚13499 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55761540 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55761248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23550441 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2053751005238608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242804713 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15763202391040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109262069 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚448806971911680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15674885763584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133552789 _runtime‚12242.519089178 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15675227187712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15674885763584 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2069402596883488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2081513946113024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23550444 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚448806990523904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133542646 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32211096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2069402239282854 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2081513946113024 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2081513587649536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242814858 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚464570192914944 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 $ _timestamp‚1.784519068942044e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2065838701885952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚13499 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55761540 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55761248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23550441 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2053751005238608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242804713 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15763202391040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _step‚1110 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133542646 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2081513587649536 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2065838718925312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32210807 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2053751022277968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2065986976388096 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109269862 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2065986993427456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2069551135939324 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2053898435906080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15676337293824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133552302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242822162 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2069550778338578 trainer/global_step‚13499 % _timestamp‚1.7845190695513995e+09 _runtime‚12243.127093737 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚464604366126592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55765208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2053898452945440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2081663330721280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133562459 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242822162 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚448840170793984 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1111 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109269860 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55765507 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109269862 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242832321 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚464604492098048 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2065986976388096 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109269862 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2065986993427456 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2069551135939324 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2053898435906080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15676337293824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133552302 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 242822162 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2069550778338578 trainer/global_step‚13499 % _timestamp‚1.7845190695513995e+09 _runtime‚12243.127093737 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚464604366126592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55765208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2053898452945440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2081663330721280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133562459 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242822162 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚448840170793984 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1111 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109269860 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55765507 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109269862 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242832321 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534×∨4§+/ _wandbruntime‚12243  _runtime‚12243ã¡Èf¨+ˆ 1loss/SRqqt1/detection_loss‚0.6305756372513567 1loss/SRqqt2/detection_loss‚0.6317514438168533 $ _timestamp‚1.784519069553451e+09 _runtime‚12243.131996886 _step‚1112 3loss/FBt2/assignment_loss‚0.0008262272068326957 /loss/FRt1/detection_loss‚0.4167729178942565 /loss/FRt2/detection_loss‚1.6152757338260133 /loss/FBt1/detection_loss‚0.3030045533013734 0loss/FBt2/detection_loss‚0.30743949460568576 3loss/SRqqt1/assignment_loss‚0.36213910248810854 1loss/FBt1/assignment_loss‚0.01008831745246767 trainer/global_step‚13499 %loss/total_loss‚7.067417948885814 epoch‚33 0loss/FRt1/assignment_loss‚1.4543164474717805 0loss/FRt2/assignment_loss‚0.7949790919438386 2loss/SRqqt2/assignment_loss‚0.5402489816272473T§f©+ %loss/total_loss‚7.067417948885814 epoch‚33 0loss/FRt1/assignment_loss‚1.4543164474717805 0loss/FRt2/assignment_loss‚0.7949790919438386 2loss/SRqqt2/assignment_loss‚0.5402489816272473 1loss/SRqqt1/detection_loss‚0.6305756372513567 1loss/SRqqt2/detection_loss‚0.6317514438168533 $ _timestamp‚1.784519069553451e+09 _runtime‚12243.131996886 _step‚1112 3loss/FBt2/assignment_loss‚0.0008262272068326957 /loss/FRt1/detection_loss‚0.4167729178942565 /loss/FRt2/detection_loss‚1.6152757338260133 /loss/FBt1/detection_loss‚0.3030045533013734 0loss/FBt2/detection_loss‚0.30743949460568576 3loss/SRqqt1/assignment_loss‚0.36213910248810854 1loss/FBt1/assignment_loss‚0.01008831745246767 trainer/global_step‚13499Øt.²€ª+‚(:õ ©³öÒ‘ÿÆË% disk./.usageGB‚1168.2004470825195 disk.dm-0.in‚ 530.234375" disk.nvme0c0n1.in‚ 300.36328125' disk.nvme1c1n1.out‚36962.2197265625 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 621.6640625# proc.memory.rssMB‚ 3024.74609375 network.recv‚ 3.198896e+06) disk./.usagePercent‚76.09187084074382" disk.dm-1.out‚14465.6005859375( disk.nvme0c0n1.out‚83592.21435546875& disk.nvme0n1.out‚83592.21435546875" disk.nvme1c1n1.in‚ 229.87109375 network.sent‚ 2.1622414e+07 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 300.36328125% disk.nvme1n1.out‚36962.2197265625 disk.md0.in‚0& proc.memory.availableMB‚ 491.046875$ disk.dm-0.out‚104794.64208984375$ memory_percent‚97.00288772583008) proc.memory.percent‚18.46158504486084 cpu‚25.688344828036346 proc.cpu.threads‚17-|ˆ“«+‚(:ˆ ©³öÒèÊŽÌ gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚168.852' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚56.284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wŜ퀬+‚(:õ ¸³öÒ÷ÏçË cpu‚25.688073075655858 network.recv‚ 3.202274e+06$ disk./.usageGB‚1166.866928100586$ disk.dm-0.out‚104795.48583984375" disk.nvme0c0n1.in‚ 300.36328125' disk.nvme0c0n1.out‚83593.6181640625% disk.nvme0n1.out‚83593.6181640625 disk.nvme1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3024.74609375 network.sent‚ 2.1673049e+07) disk./.usagePercent‚76.00501078655502 disk.md0.out‚ 622.36328125 disk.dm-0.in‚ 530.234375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 300.36328125" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.00288772583008 disk.md0.in‚0 disk.dm-1.in‚0( disk.nvme1c1n1.out‚36963.15087890625) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17" disk.dm-1.out‚14465.6513671875 disk.rbd0.in‚0& disk.nvme1n1.out‚36963.15087890625& proc.memory.availableMB‚ 491.046875ä®Cž­+‚(:“ ¸³öÒ «¬Ì gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚267.622' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.20733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[§;4®+/ _wandbruntime‚12284  _runtime‚12284÷2’̯+Æ _runtime‚12284.828011803 _step‚1113 ' lr-AdamW/pg1‚0.00015012440004196124 ' lr-AdamW/pg2‚0.00015012440004196124 trainer/global_step‚13549 % _timestamp‚1.7845191112524116e+09A7WPÑ°+Ë ' lr-AdamW/pg2‚0.00015012440004196124 trainer/global_step‚13549 % _timestamp‚1.7845191112524116e+09 _runtime‚12284.828011803 _step‚1113 ' lr-AdamW/pg1‚0.00015012440004196124ÙGC:Ò4±+/ _wandbruntime‚12284  _runtime‚122842-t8 (²+†P ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2073252426993152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15818996396544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2088982881858560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32324887 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚13549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 243689160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 243679015 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2088983240322048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15730796289536 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2076829613470496 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109651719 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2088982881858560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55962332 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450465985415680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15707057818928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚1114 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109651717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23637144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2073252426993152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55962028 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2061122555651568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2076829255869862 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109651717 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚466285000424448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15730796289536 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15730454865408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32325188 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450466004027904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _runtime‚12284.828903524 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2088983240322048 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2073252444032512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15818891127296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15818996396544 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2088982881858560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32324887 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚13549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 243689160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 243679015 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2088983240322048 _LDeviceStatsMonitor.on_train+U3¡ _batch_start/allocated_bytes.small_pool.allocated‚15730796289536 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2076829613470496 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109651719 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2088982881858560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55962332 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚450465985415680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15707057818928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚1114 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109651717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23637144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2073252426993152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55962028 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2061122555651568Ú#—§|´+‚(:ñ dzöÒ„ΨË disk.dm-1.in‚0 cpu‚25.687805243175244 network.sent‚ 2.1675971e+07) disk./.usagePercent‚76.00505277866301% disk./.usageGB‚1166.8675727844238% disk.nvme1c1n1.out‚36964.04296875" disk.dm-1.out‚14465.7998046875( disk.nvme0c0n1.out‚83596.30419921875 disk.nvme0n1.in‚ 300.36328125& disk.nvme0n1.out‚83596.30419921875 disk.nvme1n1.in‚ 229.87109375# disk.nvme1n1.out‚36964.04296875 disk.md0.out‚ 622.9609375 disk.dm-0.in‚ 530.234375 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚97.00307846069336& proc.memory.availableMB‚ 491.015625 network.recv‚ 3.205564e+06 disk.md0.in‚0" disk.nvme0c0n1.in‚ 300.36328125# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17" disk.nvme1c1n1.in‚ 229.87109375$ disk.dm-0.out‚104797.66259765625ëNÌœµ+‚(:‘ dzöÒ”ƒšÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.524' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.84133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u Ì!4¶+/ _wandbruntime‚12285  _runtime‚12285ÀÊå$'·+N ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15820009087488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134036954 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55966053 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 243696464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2076977794925586 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2061269986319040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134036954 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466319296550400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23638875 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109659510 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2089132624930304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450499184297984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2073400701495296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109659512 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2073400718534656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109659512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2076978152526332 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 243696464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845191118652427e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2089132624930304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 243706623 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23638878 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15707825645906 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32327178 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15820112252416 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109659510 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2073400701495296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚466319174773248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2073400718534656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2089132266459648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157Qï°'¸+•N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2073400718534656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2089132266459648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15820009087488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134036954 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55966053 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 243696464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2076977794925586 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2061269986319040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134036954 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466319296550400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23638875 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109659510 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2089132624930304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450499184297984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2073400701495296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109659512 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2073400718534656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109659512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2076978152526332 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 243696464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845191118652427e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2089132624930304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 243706623 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23638878 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15707825645906 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32327178 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15820112252416 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109659510 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2073400701495296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚466319174773248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528Û’s¥Ý4¹+/ _wandbruntime‚12285  _runtime‚12285BŸ¹Ûº+‰ 0loss/FRt2/assignment_loss‚0.6806695784449273 3loss/SRqqt1/assignment_loss‚0.43234260282907266 2loss/SRqqt2/assignment_loss‚0.4980682825376108 4loss/FBt2/assignment_loss‚0.00022865254740606208 0loss/FRt1/detection_loss‚0.44046639054969045 /loss/FBt2/detection_loss‚0.3864258578734819 _step‚1116 /loss/FRt1/assignment_loss‚1.498215650587531 1loss/FBt1/assignment_loss‚0.03018369569047234 1loss/SRqqt1/detection_loss‚0.6276645202849667 trainer/global_step‚13549 _runtime‚12285.443550146 1loss/SRqqt2/detection_loss‚0.6454549062082579 epoch‚33 $ _timestamp‚1.784519111868237e+09 /loss/FRt2/detection_loss‚1.5917086964044387 0loss/FBt1/detection_loss‚0.49185778808188857 %loss/total_loss‚7.323286622039745ül¼k»+Ž %loss/total_loss‚7.323286622039745 0loss/FRt2/assignment_loss‚0.6806695784449273 3loss/SRqqt1/assignment_loss‚0.43234260282907266 2loss/SRqqt2/assignment_loss‚0.4980682825376108 4loss/FBt2/assignment_loss‚0.00022865254740606208 0loss/FRt1/detection_loss‚0.44046639054969045 /loss/FBt2/detection_loss‚0.3864258578734819 _step‚1116 /loss/FRt1/assignment_loss‚1.498215650587531 1loss/FBt1/assignment_loss‚0.03018369569047234 1loss/SRqqt1/detection_loss‚0.6276645202849667 trainer/global_step‚13549 _runtime‚12285.443550146 1loss/SRqqt2/detection_loss‚0.6454549062082579 epoch‚33 $ _timestamp‚1.784519111868237e+09 /loss/FRt2/detection_loss‚1.5917086964044387 0loss/FBt1/detection_loss‚0.49185778808188857Ü°£~¼+‚(:ó Ö³öÒÞ òË" disk.dm-1.out‚14465.9013671875& disk.nvme0c0n1.out‚83597.248046875$ disk.nvme0n1.out‚83597.248046875 disk.md0.in‚0$ memory_percent‚97.00307846069336# proc.memory.rssMB‚ 3024.74609375 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 623.27734375 disk.dm-0.in‚ 530.234375& proc.memory.availableMB‚ 491.015625 network.recv‚ 3.209727e+06 disk.dm-1.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 300.36328125 disk.nvme0n1.in‚ 300.36328125$ disk.dm-0.out‚104798.33447265625) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17 network.sent‚ 2.1741745e+07( disk.nvme1c1n1.out‚36964.58056640625& disk.nvme1n1.out‚36964.58056640625 cpu‚25.687513107235063( disk./.usagePercent‚76.0050381186963% disk./.usageGB‚1166.8673477172852P³ŠŸž½+‚(:“ Ö³öÒ”¾Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.117' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.37233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊGjz¾+‚(:ï å³öÒ£€ÑË disk.dm-1.in‚0 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375# disk.nvme1n1.out‚36965.47265625$ memory_percent‚97.00307846069336 proc.cpu.threads‚17 network.recv‚ 3.213855e+06) disk./.usagePercent‚76.00508408638848 disk.dm-0.in‚ 530.234375# disk.dm-1.out‚14466.04150390625& proc.memory.availableMB‚ 491.015625# proc.memory.rssMB‚ 3024.74609375 cpu‚25.687286146870864% disk./.usageGB‚1166.8680534362793" disk.nvme0c0n1.in‚ 300.36328125& disk.nvme0c0n1.out‚83601.416015625$ disk.dm-0.out‚104802.19580078125) proc.memory.percent‚18.46158504486084 network.sent‚ 2.1781835e+07 disk.rbd0.in‚0 disk.nvme0n1.in‚ 300.36328125$ disk.nvme0n1.out‚83601.416015625% disk.nvme1c1n1.out‚36965.47265625 disk.md0.in‚0 disk.md0.out‚ 623.76953125c«-;ž¿+‚(:“ å³öÒª¯—Ì gpu.0.gpu‚92.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚274.631' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.54366666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ROŒ4À+/ _wandbruntime‚12326  _runtime‚12326y®ÌÁ+Æ ' lr-AdamW/pg1‚0.00014801345124514691 ' lr-AdamW/pg2‚0.00014801345124514691 trainer/global_step‚13599 % _timestamp‚1.7845191532755225e+09 _runtime‚12326.851109542 _step‚1117ø©[ÑÂ+Ë _step‚1117 ' lr-AdamW/pg1‚0.00014801345124514691 ' lr-AdamW/pg2‚0.00014801345124514691 trainer/global_step‚13599 % _timestamp‚1.7845191532755225e+09 _runtime‚12326.851109542Ýy@} 4Ã+/ _wandbruntime‚12326  _runtime‚12326ˆò¸¼ÔÄ+†P MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32440023 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110041367 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2068494089025168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚13599 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2096452176067584 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2084256630057504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110041369 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15874676340736 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2084256272456870 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134511950 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244553317 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2080666169139712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚467999675260416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_s9Ž¤å8tart/inactive_split.all.freed‚56163577 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚1118 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23723844 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 115755008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134522093 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15786365391360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2096452176067584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110041367†xEÏ(Å+‹P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2080666152100352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚452124998919680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23723841 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110041369 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15786023967232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2080666152100352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134522093 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2096452534531072 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2096452176067584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110041367 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32440023 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110041367 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2068494089025168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚13599 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2096452176067584 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2084256630057504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110041369 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15874676340736 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2084256272456870 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134511950 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244553317 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2080666169139712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚467999675260416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56163577 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 171722240 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚1118 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23723844 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 115755008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134522093 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15786365391360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56167623 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2080814426602496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15787134066176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32442335 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15875788412928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244580925 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 % _timestamp‚1.7845191538901255e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2080814426602496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚13599 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134531763 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110049160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110049162 _runtime‚12327.465893096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15763308859314 _step‚1119 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15787475497472 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134531763 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32442048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2096601560668672 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244580925 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452158179189760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110049160 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2096601919139328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244570766 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15787475497472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110049162 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2080814443641856 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚468034101962752 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2068641519692640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134521606 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15787134066176 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15787134066176 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56167623 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2080814426602496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15787134066176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32442335 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15875788412928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244580925 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 % _timestamp‚1.7845191538901255e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2080814426602496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚13599 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134531763 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110049160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110049162 _runtime‚12327.465893096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15763308859314 _step‚1119 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15787475497472 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134531763 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32442048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2096601560668672 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244580925 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452158179189760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110049160 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2096601919139328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244570766 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15787475497472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110049162 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 171722240 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2080814443641856 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚468034101962752 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2068641519692640ßË‘îI4É+/ _wandbruntime‚12327  _runtime‚12327oEdÀÊ+ˆ $loss/total_loss‚6.78703545649186 trainer/global_step‚13599 % _timestamp‚1.7845191538910925e+09 0loss/FRt1/assignment_loss‚1.5217633248516247 1loss/SRqqt1/assignment_loss‚0.429183111281554 3loss/SRqqt2/assignment_loss‚0.43909996511729077 0loss/FRt2/assignment_loss‚0.6029305417397443 2loss/FBt1/assignment_loss‚0.006398834771437048 4loss/FBt2/assignment_loss‚0.00015990511304961628 /loss/FRt2/detection_loss‚1.3271224006385927 1loss/SRqqt2/detection_loss‚0.6088540548337089 /loss/FBt2/detection_loss‚0.3877582833895253 _step‚1120 /loss/FRt1/detection_loss‚0.4605313723287548 /loss/FBt1/detection_loss‚0.3639690804718013 epoch‚33 _runtime‚12327.466408087 1loss/SRqqt1/detection_loss‚0.6392645819547764 §ÝË+ 0loss/FRt2/assignment_loss‚0.6029305417397443 2loss/FBt1/assignment_loss‚0.006398834771437048 4loss/FBt2/assignment_loss‚0.00015990511304961628 /loss/FRt2/detection_loss‚1.3271224006385927 1loss/SRqqt2/detection_loss‚0.6088540548337089 /loss/FBt2/detection_loss‚0.3877582833895253 _step‚1120 /loss/FRt1/detection_loss‚0.4605313723287548 /loss/FBt1/detection_loss‚0.3639690804718013 epoch‚33 _runtime‚12327.466408087 1loss/SRqqt1/detection_loss‚0.6392645819547764 $loss/total_loss‚6.78703545649186 trainer/global_step‚13599 % _timestamp‚1.7845191538910925e+09 0loss/FRt1/assignment_loss‚1.5217633248516247 1loss/SRqqt1/assignment_loss‚0.429183111281554 3loss/SRqqt2/assignment_loss‚0.43909996511729077àSYºƒÌ+‚(:ø ô³öÒ‚€òË) disk./.usagePercent‚76.00508657112861& disk.nvme0n1.out‚83602.24169921875 disk.dm-0.in‚ 530.234375# disk.dm-1.out‚14466.06884765625# proc.memory.rssMB‚ 3024.74609375 cpu‚25.687502957647727 network.sent‚ 2.17976e+07 network.recv‚ 3.217383e+06" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚36966.28271484375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚36966.28271484375$ disk.dm-0.out‚104803.14501953125( proc.memory.availableMB‚ 490.94921875$ disk./.usageGB‚1166.868091583252" disk.nvme0c0n1.in‚ 300.36328125 disk.md0.in‚0 disk.md0.out‚ 624.06640625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚97.00348377227783( disk.nvme0c0n1.out‚83602.24169921875 disk.nvme0n1.in‚ 300.36328125) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17/dAbžÍ+‚(:“ ô³öÒ¾²¶Ì gpu.0.gpu‚97.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.597' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.53233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U|úc4Î+/ _wandbruntime‚12343  _runtime‚12343^KðÄ<Ï+¾y :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20637564361095428 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26936665177345276 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6957363486289978 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 *jet/accuracy_2_of_3‚0.5521536469459534 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05221063643693924 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 epoch‚33 trainer/global_step‚13599 _runtime‚12343.764831523 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21547503769397736 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2807888388633728 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8575775623321533 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3539491593837738 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34882721304893494 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7185136675834656 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9205209016799927 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07327276468276978 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 6validation_average_jet_accuracy‚0.3764449656009674 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5677083134651184 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6988471150398254 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 *jet/accuracy_1_of_3‚0.9198514819145203 particle/accuracy_2_of_4‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9983728528022766 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7169834971427917 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6284635663032532 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.16047541797161102 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22795304656028748 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3137508034706116 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3021681010723114 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8596749901771545 /particle/accuracy_4_of_4‚0.9487894177436829 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4936649799346924 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2798991799354553 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8577097654342651 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27250242233276367 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6349055171012878 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.605668306350708 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06475190073251724 jet/accuracy_2_of_5‚1 )loss/val_total_loss‚7.442644813158536 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6567526459693909 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6602144241333008 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2678413391113281 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.465882807970047 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2553650736808777 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6272555589675903 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5860344171524048 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.626101016998291 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_4‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861111044883728 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 *jet/accuracy_1_of_2‚0.7675284147262573 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9522351622581482 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5724964737892151 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23827634751796722 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.29357072710990906 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3065114915370941 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9982896447181702 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3021681010723114 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7682291865348816 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7024432420730591 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34729498624801636 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.602203369140625 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6782606840133667 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8744818568229675 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1244191974401474 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2775055468082428 $ _timestamp‚1.784519170188785e+09 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30270370841026306 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2433333396911621 particle/accuracy_1_of_2‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6792284846305847 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4920516312122345 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6555451154708862 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3111236095428467 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06582962721586227 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04080826789140701 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6106407642364502 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6448437571525574 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06535515189170837 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 (particle/accuracy‚0.8544729351997375 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7214075326919556 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6289741396903992 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8899669647216797 & Purity/0FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4524414539337158 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2904936969280243 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.601215124130249 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_3‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3578648865222931 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9656561613082886 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31515032052993774 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8672335743904114 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6486401557922363 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 +particle/specificity‚0.9118072986602783 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6042466759681702 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6725309491157532 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6094136238098145 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_2_of_4‚0.7324236631393433 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9987468719482422 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5334039330482483 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8537226319313049 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3286389410495758 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7809737324714661 /particle/accuracy_3_of_3‚0.9994120597839355 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21245667338371277 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_1‚0.3380112946033478 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04918835312128067 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9931437373161316 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6290221214294434 particle/accuracy_1_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6083754897117615 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22616541385650635 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9942034482955933 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6091666221618652 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_3‚1 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.593453586101532 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7809737324714661 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6266568303108215 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7075676918029785 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04918835312128067 particle/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5853095054626465 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7020449042320251 5 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.30706587433815 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.665022075176239 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.717595636844635 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5853095054626465 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07242327183485031 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10199564695358276 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 _step‚1121 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7070517539978027 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5842416882514954 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.053184207528829575 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.468548446893692 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7777588963508606 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2019912600517273 +jet/accuracy_3_of_3‚0.07713009417057037 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20255978405475616 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.767393946647644 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3135974705219269 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6489005088806152 +jet/accuracy_3_of_4‚0.09558874368667603éÑ“€‚(Ð+Ãy 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2807888388633728 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8575775623321533 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3539491593837738 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34882721304893494 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7185136675834656 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9205209016799927 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07327276468276978 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 6validation_average_jet_accuracy‚0.3764449656009674 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5677083134651184 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6988471150398254 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 *jet/accuracy_1_of_3‚0.9198514819145203 particle/accuracy_2_of_4‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9983728528022766 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7169834971427917 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6284635663032532 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.16047541797161102 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22795304656028748 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3137508034706116 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.3021681010723114 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8596749901771545 /particle/accuracy_4_of_4‚0.9487894177436829 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4936649799346924 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2798991799354553 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8577097654342651 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27250242233276367 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6349055171012878 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.605668306350708 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06475190073251724 jet/accuracy_2_of_5‚1 )loss/val_total_loss‚7.442644813158536 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6567526459693909 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6602144241333008 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2678413391113281 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.465882807970047 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2553650736808777 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6272555589675903 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5860344171524048 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.626101016998291 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_4‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861111044883728 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 *jet/accuracy_1_of_2‚0.7675284147262573 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9522351622581482 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5724964737892151 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23827634751796722 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.29357072710990906 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/0FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3065114915370941 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9982896447181702 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.3021681010723114 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7682291865348816 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7024432420730591 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.34729498624801636 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.602203369140625 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6782606840133667 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8744818568229675 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1244191974401474 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2775055468082428 $ _timestamp‚1.784519170188785e+09 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.30270370841026306 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2433333396911621 particle/accuracy_1_of_2‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6792284846305847 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4920516312122345 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 'particle/f_score‚0.6555451154708862 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3111236095428467 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06582962721586227 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04080826789140701 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6106407642364502 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6448437571525574 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06535515189170837 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 (particle/accuracy‚0.8544729351997375 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7214075326919556 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6289741396903992 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8899669647216797 & Purity/0FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4524414539337158 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2904936969280243 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.601215124130249 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_3‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3578648865222931 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9656561613082886 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.31515032052993774 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8672335743904114 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6486401557922363 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 +particle/specificity‚0.9118072986602783 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6042466759681702 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6725309491157532 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6094136238098145 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_2_of_4‚0.7324236631393433 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9987468719482422 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5334039330482483 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8537226319313049 7JêG Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3286389410495758 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7809737324714661 /particle/accuracy_3_of_3‚0.9994120597839355 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21245667338371277 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_1‚0.3380112946033478 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04918835312128067 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9931437373161316 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6290221214294434 particle/accuracy_1_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6083754897117615 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22616541385650635 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9942034482955933 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6091666221618652 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_3‚1 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.593453586101532 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7809737324714661 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6266568303108215 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7075676918029785 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04918835312128067 particle/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5853095054626465 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.7020449042320251 5 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.30706587433815 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.665022075176239 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.717595636844635 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5853095054626465 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07242327183485031 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.10199564695358276 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 _step‚1121 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7070517539978027 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5842416882514954 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.053184207528829575 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.468548446893692 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7777588963508606 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2019912600517273 +jet/accuracy_3_of_3‚0.07713009417057037 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20255978405475616 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.767393946647644 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3135974705219269 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6489005088806152 +jet/accuracy_3_of_4‚0.09558874368667603 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20637564361095428 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26936665177345276 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6957363486289978 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 *jet/accuracy_2_of_3‚0.5521536469459534 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05221063643693924 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 epoch‚33 trainer/global_step‚13599 _runtime‚12343.764831523 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21547503769397736 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0á :U…Ñ+‚(:ú ƒ´öÒî¨ÃË) disk./.usagePercent‚76.00508781349866% disk./.usageGB‚1166.8681106567383" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚36967.23876953125 disk.nvme1n1.in‚ 229.87109375( disk.nvme0c0n1.out‚83604.80419921875 disk.nvme0n1.in‚ 300.36328125# proc.memory.rssMB‚ 3024.74609375 disk.dm-0.in‚ 530.234375$ disk.dm-0.out‚104805.50537109375 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 300.36328125( proc.memory.availableMB‚ 490.91796875 cpu‚25.690270251224103 disk.dm-1.in‚0 network.sent‚ 2.1860052e+07 network.recv‚ 3.221791e+06 disk.md0.out‚ 624.5234375 disk.rbd0.out‚0$ memory_percent‚97.00367450714111) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17& disk.nvme1n1.out‚36967.23876953125 disk.md0.in‚0# disk.dm-1.out‚14466.23681640625& disk.nvme0n1.out‚83604.80419921875eÑØJžÒ+‚(:“ ƒ´öÒèË‹Ì gpu.0.gpu‚55.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚278.438' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.81266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÑ^tÓ+‚(:é ’´öÒë¿¥Ë& proc.memory.availableMB‚ 490.921875# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 network.sent‚ 2.1905879e+07! disk.nvme1n1.out‚ 36967.765625 disk.dm-0.in‚ 530.234375" disk.nvme0c0n1.in‚ 300.36328125 cpu‚25.69002045881955 proc.cpu.threads‚17 disk.nvme1n1.in‚ 229.87109375# disk.dm-1.out‚14466.33837890625' disk.nvme0c0n1.out‚83605.7724609375# disk.nvme1c1n1.out‚ 36967.765625 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚76.00508756502465$ disk./.usageGB‚1166.868106842041 disk.md0.out‚ 624.8359375$ disk.dm-0.out‚104806.21240234375 disk.dm-1.in‚0 disk.nvme0n1.in‚ 300.36328125 network.recv‚ 3.226456e+06 disk.md0.in‚0% disk.nvme0n1.out‚83605.7724609375" disk.nvme1c1n1.in‚ 229.87109375# memory_percent‚97.0036506652832¨áÙžÔ+‚(:“ ’´öÒêÍæË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚126.734' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚42.24466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘¸jzÕ+‚(:ï ¡´öÒµÔßË) proc.memory.percent‚18.46158504486084 cpu‚25.68974810805333 proc.cpu.threads‚17 network.sent‚ 2.1910707e+07 disk.rbd0.out‚0 disk.md0.in‚0 disk.md0.out‚ 625.15625$ disk.dm-0.out‚104808.30126953125# memory_percent‚97.0036506652832& proc.memory.availableMB‚ 490.921875 disk.rbd0.in‚0$ disk.nvme0n1.out‚83608.005859375" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚36968.39306640625 disk.dm-0.in‚ 530.234375 network.recv‚ 3.229577e+06% disk./.usageGB‚1166.8681945800781 disk.dm-1.in‚0" disk.dm-1.out‚14466.4306640625" disk.nvme0c0n1.in‚ 300.36328125& disk.nvme0c0n1.out‚83608.005859375 disk.nvme1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3024.74609375) disk./.usagePercent‚76.00509327992692 disk.nvme0n1.in‚ 300.36328125& disk.nvme1n1.out‚36968.39306640625T´ÁžÖ+‚(:“ ¡´öÒŒ¡ Ì gpu.0.gpu‚54.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚279.238' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.07933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W{ 4×+/ _wandbruntime‚12385  _runtime‚12385mïÔ‡ÌØ+Æ ' lr-AdamW/pg1‚0.00014591272957459493 ' lr-AdamW/pg2‚0.00014591272957459493 trainer/global_step‚13649 % _timestamp‚1.7845192116610558e+09 _runtime‚12385.238028892 _step‚1122Ú@©ÑÙ+Ë _runtime‚12385.238028892 _step‚1122 ' lr-AdamW/pg1‚0.00014591272957459493 ' lr-AdamW/pg2‚0.00014591272957459493 trainer/global_step‚13649 % _timestamp‚1.7845192116610558e+09âŸå¾4Ú+/ _wandbruntime‚12385  _runtime‚12385RAÙô (Û+…P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15862252328960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2093625206501633 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110553827 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135088239 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _runtime‚12385.238093362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56402503 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2090022238122496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56402807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 245631921 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110553827 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2090022221083136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15862252328960 $ _timestamp‚1.784519211661644e+09 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15837987533367 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135078096 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2105884131987968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚469757778701312 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32587846 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23814657 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15951299175936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15861910904832 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110553825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1123 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2105884131987968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚453806591086080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _runtime‚12385.238093362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56402503 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2090022238122496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56402807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 245631921 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110553827 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2090022221083136 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15862252328960 $ _timestamp‚1.784519211661644e+09 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15837987533367 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135078096 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2105884131987968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚469757778701312 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32587846 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23814657 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15951299175936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15861910904832 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110553825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1123 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2105884131987968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚453806591086080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135087752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2093773745557469 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚453839789968384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚453839771356160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15863362435072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15839095921619 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15863021003776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245659529 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 245649370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2090170495585280 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110561620 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110561618 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 245659529 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2090170495585280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚12385.851933898 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2106033516589056 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135087752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2106033875059712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32590444 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23816391 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110561620 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56406838 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23816394 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚469792078973440 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _step‚1124 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56406537 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135097909 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2090170512624640Ó½]'ß+”N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110561620 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56406838 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23816394 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚469792078973440 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _step‚1124 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56406537 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135097909 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2090170512624640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135097909 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2093773387956723 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135087752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2093773745557469 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚453839789968384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚453839771356160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15863362435072 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15839095921619 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15863021003776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245659529 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 245649370 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2090170495585280 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110561620 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110561618 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 245659529 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2090170495585280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚12385.851933898 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2106033516589056 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135087752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2106033875059712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32590444 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23816391 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272ä(…L(4à+/ _wandbruntime‚12385  _runtime‚12385ëJâá+‰ /loss/FRt1/detection_loss‚0.5277101072188455 1loss/SRqqt1/detection_loss‚0.6671104885258546 %loss/total_loss‚7.329807828970591 epoch‚34 trainer/global_step‚13649 % _timestamp‚1.7845192122776418e+09 /loss/FRt2/detection_loss‚1.4859536360663212 1loss/SRqqt2/detection_loss‚0.6286444829655375 /loss/FBt1/detection_loss‚0.5141181749895045 3loss/SRqqt2/assignment_loss‚0.48693152721526145 3loss/FBt1/assignment_loss‚0.0004518564253847407 2loss/FBt2/assignment_loss‚0.016418724295158397 0loss/FBt2/detection_loss‚0.47614801096041565 _runtime‚12385.85301829 0loss/FRt1/assignment_loss‚1.4944500990316902 0loss/FRt2/assignment_loss‚0.6654359186218738 2loss/SRqqt1/assignment_loss‚0.3664348026547423 _step‚1125쮊2â+Ž 0loss/FRt1/assignment_loss‚1.4944500990316902 0loss/FRt2/assignment_loss‚0.6654359186218738 2loss/SRqqt1/assignment_loss‚0.3664348026547423 _step‚1125 /loss/FRt1/detection_loss‚0.5277101072188455 1loss/SRqqt1/detection_loss‚0.6671104885258546 %loss/total_loss‚7.329807828970591 epoch‚34 trainer/global_step‚13649 % _timestamp‚1.7845192122776418e+09 /loss/FRt2/detection_loss‚1.4859536360663212 1loss/SRqqt2/detection_loss‚0.6286444829655375 /loss/FBt1/detection_loss‚0.5141181749895045 3loss/SRqqt2/assignment_loss‚0.48693152721526145 3loss/FBt1/assignment_loss‚0.0004518564253847407 2loss/FBt2/assignment_loss‚0.016418724295158397 0loss/FBt2/detection_loss‚0.47614801096041565 _runtime‚12385.85301829åiÜTIyã+‚(:î °´öÒ³Ÿ Ë proc.cpu.threads‚17" disk.dm-1.out‚14466.4697265625 disk.nvme0n1.in‚ 300.36328125$ disk.dm-0.out‚104808.85205078125& proc.memory.availableMB‚ 490.890625) proc.memory.percent‚18.46158504486084 cpu‚25.689480536642375" disk.nvme0c0n1.in‚ 300.36328125# disk.nvme0n1.out‚83608.68359375( disk.nvme1c1n1.out‚36968.66845703125& disk.nvme1n1.out‚36968.66845703125 network.sent‚ 2.1913629e+07) disk./.usagePercent‚76.00509576466705 disk.dm-1.in‚0% disk.nvme0c0n1.out‚83608.68359375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚625.3125 network.recv‚ 3.233035e+06% disk./.usageGB‚1166.8682327270508 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 530.234375$ memory_percent‚97.00384140014648# proc.memory.rssMB‚ 3024.74609375q3fžä+‚(:“ °´öÒÈ¢•Ì gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚261.636' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.21200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0» d3“å+‚(:ˆ ¿´öÒ‰ÅîË gpu.0.gpu‚54.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚285.939' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.313 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|k–‚æ+‚(:÷ ¿´öÒ…áÌ network.sent‚ 2.1977199e+07 network.recv‚ 3.237028e+06% disk./.usageGB‚1169.5403022766113 disk.dm-0.in‚ 530.7578125 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚97.0059871673584 cpu‚25.689245201652298 disk.nvme0n1.in‚ 300.88671875" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 627.78515625# disk.dm-0.out‚105894.4853515625 disk.dm-1.in‚0# disk.dm-1.out‚14467.27392578125' proc.memory.availableMB‚ 490.5390625 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 300.88671875( disk.nvme0c0n1.out‚83631.01513671875' disk.nvme1c1n1.out‚38041.0537109375 disk.md0.in‚0) proc.memory.percent‚18.46158504486084) disk./.usagePercent‚76.17914361026617& disk.nvme0n1.out‚83631.01513671875 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚38041.3037109375# proc.memory.rssMB‚ 3024.74609375³6v}ç+‚(:ò δöÒ‹è·Ë network.sent‚ 2.1982839e+07) disk./.usagePercent‚76.17892942566803 disk.md0.in‚0# disk.dm-1.out‚14467.38330078125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 308.64453125% disk.nvme1c1n1.out‚39694.19921875' proc.memory.availableMB‚ 490.2890625 disk.dm-0.in‚ 538.515625 disk.dm-1.in‚0( disk.nvme0c0n1.out‚83637.72119140625) proc.memory.percent‚18.46158504486084 network.recv‚ 3.240439e+06# disk.dm-0.out‚107554.2646484375 disk.rbd0.in‚0& disk.nvme0n1.out‚83637.72119140625" disk.nvme1c1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3024.74609375 cpu‚25.68896234682312 proc.cpu.threads‚17% disk./.usageGB‚1169.5370140075684 disk.md0.out‚ 629.36328125 disk.nvme0n1.in‚ 308.64453125 disk.nvme1n1.in‚ 229.87109375# disk.nvme1n1.out‚39694.19921875$ memory_percent‚97.00751304626465í]¼<è+‚(:’ δöÒ–‰øË gpu.0.gpu‚92.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚279.53' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.17666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð4é+/ _wandbruntime‚12427  _runtime‚12427Ñó?Ìê+Æ _step‚1126 ' lr-AdamW/pg1‚0.00014382236995573244 ' lr-AdamW/pg2‚0.00014382236995573244 trainer/global_step‚13699 % _timestamp‚1.7845192540943768e+09 _runtime‚12427.671059014åb$Ñë+Ë ' lr-AdamW/pg1‚0.00014382236995573244 ' lr-AdamW/pg2‚0.00014382236995573244 trainer/global_step‚13699 % _timestamp‚1.7845192540943768e+09 _runtime‚12427.671059014 _step‚1126æK-×E4ì+/ _wandbruntime‚12427  _runtime‚12427ùCÎn#í+…P QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2113353784660480 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2101052223088641 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135572891 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23901360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚471472792627712 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2097435946190336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15917480006656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110943477 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135572891 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16007169425408 _step‚1127 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2113353426196992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32703015 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110943477 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16007064156160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15893811308049 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2097435963229696 LBDeviceStatsMonitor.on_trai±na¢n_batch_start/allocation.small_pool.peak‚12303 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2101051865488007 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 $ _timestamp‚1.784519254095054e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15917480006656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246506223 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2097435946190336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135562748 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚2721pØ(î+ŠP ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2113353784660480 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2101052223088641 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135572891 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23901360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚471472792627712 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2097435946190336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15917480006656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110943477 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135572891 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16007169425408 _step‚1127 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2113353426196992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32703015 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110943477 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16007064156160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15893811308049 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2097435963229696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2101051865488007 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 $ _timestamp‚1.784519254095054e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15917480006656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246506223 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2097435946190336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135562748 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634ç­O4ï+/ _wandbruntime‚12428  _runtime‚12428Ú9’Ð'ð+ŽN C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15894919696413 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2113503169268736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚471506958778368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16008173918208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2085305842448064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246533831 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135572404 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15918931536896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2101200762144477 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15894579135027 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2097584237731840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246533831 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚13699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2097584237731840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56608375 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚1128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56608668 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32705284 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2113502810798080 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23903091 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2101200404543731 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚455498784860160 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚12428.27883294 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135582561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2113503169268736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246523672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135582561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2097584220692480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110951270 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23903094 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15918590105600 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32705574 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2085305825408704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16008287568896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0]w;ä'ñ+“N [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2097584220692480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110951270 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23903094 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15918590105600 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32705574 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2085305825408704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16008287568896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15894919696413 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2113503169268736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚471506958778368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16008173918208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2085305842448064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246533831 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135572404 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15918931536896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2101200762144477 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15894579135027 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2097584237731840 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246533831 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚13699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2097584237731840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56608375 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚1128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56608668 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32705284 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2113502810798080 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23903091 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2101200404543731 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚455498784860160 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 _runtime‚12428.27883294 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135582561 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2113503169268736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246523672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135582561 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360è{Þº4ò+/ _wandbruntime‚12428  _runtime‚12428Ó!émó+ˆ 3loss/FBt1/assignment_loss‚0.0011972264392501625 1loss/SRqqt2/detection_loss‚0.6188075298753878 /loss/FBt2/detection_loss‚0.5601729809331071 %loss/total_loss‚7.543142491271742 epoch‚34 0loss/FRt2/assignment_loss‚0.7719435668516276 .loss/FRt2/detection_loss‚1.490691550792147 1loss/SRqqt1/detection_loss‚0.6346342919247323 trainer/global_step‚13699 % _timestamp‚1.7845192547070847e+09 _runtime‚12428.282352645 0loss/FRt1/assignment_loss‚1.5777484345342783 2loss/SRqqt1/assignment_loss‚0.4331562552142666 1loss/SRqqt2/assignment_loss‚0.471174857459832 4loss/FBt2/assignment_loss‚0.00038061526088328215 /loss/FRt1/detection_loss‚0.4647466794333775 /loss/FBt1/detection_loss‚0.5184885025528524 _step‚1129¥.ˆ8xô+ 4loss/FBt2/assignment_loss‚0.00038061526088328215 /loss/FRt1/detection_loss‚0.4647466794333775 /loss/FBt1/detection_loss‚0.5184885025528524 _step‚1129 3loss/FBt1/assignment_loss‚0.0011972264392501625 1loss/SRqqt2/detection_loss‚0.6188075298753878 /loss/FBt2/detection_loss‚0.5601729809331071 %loss/total_loss‚7.543142491271742 epoch‚·µDú›34 0loss/FRt2/assignment_loss‚0.7719435668516276 .loss/FRt2/detection_loss‚1.490691550792147 1loss/SRqqt1/detection_loss‚0.6346342919247323 trainer/global_step‚13699 % _timestamp‚1.7845192547070847e+09 _runtime‚12428.282352645 0loss/FRt1/assignment_loss‚1.5777484345342783 2loss/SRqqt1/assignment_loss‚0.4331562552142666 1loss/SRqqt2/assignment_loss‚0.471174857459832é€ðc¤|õ+‚(:ñ Ý´öÒš£üË$ disk.dm-0.out‚107560.52685546875" disk.nvme0c0n1.in‚ 309.32421875% disk.nvme0c0n1.out‚83644.32421875) proc.memory.percent‚18.46158504486084 cpu‚25.68871230355095 disk.rbd0.in‚0' proc.memory.availableMB‚ 490.0859375 network.recv‚ 3.243806e+06" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 630.234375" disk.dm-1.out‚14467.7978515625# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17 network.sent‚ 2.1998705e+07 disk.nvme0n1.in‚ 309.32421875# disk.nvme0n1.out‚83644.32421875& disk.nvme1n1.out‚39696.11083984375 disk.dm-0.in‚ 539.1953125 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚97.00875282287598) disk./.usagePercent‚76.17892172297366% disk./.usageGB‚1169.5368957519531( disk.nvme1c1n1.out‚39696.11083984375ÖÙžö+‚(:“ Ý´öÒÝÂÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.987' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.99566666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ[/÷+‚(:ô ì´öÒ–¼ÌË) disk./.usagePercent‚76.17884718077012 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.32421875% disk.nvme0n1.out‚83645.3447265625( disk.nvme1c1n1.out‚39710.73291015625 disk.nvme1n1.in‚ 229.87109375) proc.memory.percent‚18.46158504486084 cpu‚25.68846679340632 network.recv‚ 3.247882e+06 disk.nvme0n1.in‚ 309.32421875 disk.md0.in‚0 disk.md0.out‚630.4375$ disk.dm-0.out‚107575.69482421875$ memory_percent‚97.00982570648193# proc.memory.rssMB‚ 3024.74609375% disk./.usageGB‚1169.5357513427734" disk.dm-1.out‚14467.8251953125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 539.1953125( proc.memory.availableMB‚ 489.91015625 proc.cpu.threads‚17 network.sent‚ 2.2050038e+07' disk.nvme0c0n1.out‚83645.3447265625" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚39710.73291015625äÌÃü“ø+‚(:ˆ ì´öÒ·Ì gpu.0.gpu‚60.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚283.347' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.449 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áV-_{ù+‚(:ð û´öÒœ§¡Ë' disk.nvme1c1n1.out‚39711.6103515625$ disk.dm-0.out‚107578.10888671875 disk.dm-1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚39711.6103515625 disk.rbd0.in‚0$ memory_percent‚97.00984954833984) proc.memory.percent‚18.46158504486084 cpu‚25.688201706412787 network.recv‚ 3.25089e+06$ disk.nvme0n1.out‚83647.962890625 disk.md0.in‚0 disk.md0.out‚ 630.8515625 disk.dm-0.in‚ 539.1953125" disk.nvme0c0n1.in‚ 309.32421875# proc.memory.rssMB‚ 3024.74609375 network.sent‚ 2.2053061e+07) disk./.usagePercent‚76.17912075065708$ disk./.usageGB‚1169.539951324463 disk.nvme0n1.in‚ 309.32421875# disk.dm-1.out‚14468.00634765625& disk.nvme0c0n1.out‚83647.962890625% proc.memory.availableMB‚ 489.90625Êÿ’Wœú+‚(:‘ û´öÒØ÷õË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.576' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.52533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09‹²G4û+/ _wandbruntime‚12469  _runtime‚12469œ%Á®Êü+Ä trainer/global_step‚13749 % _timestamp‚1.7845192960807602e+09 _runtime‚12469.656338834 _step‚1130 & lr-AdamW/pg1‚0.0001417425066484518 & lr-AdamW/pg2‚0.0001417425066484518+\Î Ïý+É % _timestamp‚1.7845192960807602e+09 _runtime‚12469.656338834 _step‚1130 & lr-AdamW/pg1‚0.0001417425066484518 & lr-AdamW/pg2‚0.0001417425066484518 trainer/global_step‚13749êdº•4þ+/ _wandbruntime‚12469  _runtime‚12469ý«´– (ÿ+†P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111333125 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136057543 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16062877541376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 trainer/global_step‚13749 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚457124618094080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚12469.660003233 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247390670 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845192960816445e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2092529928114832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1131 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15973390532608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚457124636706304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136057543 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15973049108480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23988057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16062980713472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2120822720406016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2092529945154192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136047400 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2104849671297536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2120822720406016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32817650 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚473187495635456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15973049108480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15973390532608 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2104849688336896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56805408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473187617419776 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2120822720406016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32817650 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚473187495635456 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15973049108480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15973390532608 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2104849688336896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56805408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473187617419776 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚457124618094080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚12469.660003233 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247390670 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845192960816445e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2092529928114832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1131 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15973390532608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚457124636706304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136057543 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15973049108480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23988057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16062980713472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2120822720406016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2092529945154192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136047400 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2104849671297536ëúÊ‚a4,/ _wandbruntime‚12470  _runtime‚12470þÿ>ög‚,N QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2120972463477760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111340920 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111340918 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56809381 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247408133 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23989791 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15974500638720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247397974 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2108627778731485 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2108627421130739 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111340920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚302 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2104997962839040 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15974500638720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚457157798364160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136067213 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473221912719872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15950062348435 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473221788845568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32819892 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚13749 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136067213 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2120972463477760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚§“.¿¯ 1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23989794 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247397974 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56809381 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247408133 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23989791 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15974500638720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247397974 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2108627778731485 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2108627421130739 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111340920 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚302 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2104997962839040 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15974500638720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚457157798364160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136067213 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473221912719872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15950062348435 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473221788845568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32819892 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚13749 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136067213 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2120972463477760 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23989794 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247397974 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FžŠ,‚(:“ ™µöÒÒÓþË gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚282.961' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.32033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­=«¿‚‹,‚(:÷ ¨µöÒìÿÈË" disk.dm-1.out‚14468.2060546875 disk.rbd0.in‚0# proc.memory.rssMB‚ 3024.74609375 cpu‚25.68742446790474 disk.dm-1.in‚0 network.recv‚ 3.261632e+06% disk./.usageGB‚1169.7082214355469" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚39713.15380859375 disk.md0.in‚0 disk.dm-0.in‚ 539.1953125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚83656.7626953125 disk.nvme0n1.in‚ 309.32421875% disk.nvme0n1.out‚83656.7626953125( proc.memory.availableMB‚ 489.86328125) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17 disk.md0.out‚ 631.7109375$ disk.dm-0.out‚107586.39306640625" disk.nvme0c0n1.in‚ 309.32421875$ memory_percent‚97.01011180877686 network.sent‚ 2.2124066e+07) disk./.usagePercent‚76.19008118779006 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚39713.15380859375?`åežŒ,‚(:“ ¨µöÒïï¤Ì gpu.0.gpu‚41.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.177' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.39233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]– ¬4,/ _wandbruntime‚12512  _runtime‚12512™èÁ~ËŽ,Å $ _timestamp‚1.784519338535159e+09 _runtime‚12512.112045973 _step‚1134 ' lr-AdamW/pg1‚0.00013967327323848662 ' lr-AdamW/pg2‚0.00013967327323848662 trainer/global_step‚13799JÅÚØÐ,Ê ' lr-AdamW/pg1‚0.00013967327323848662 ' lr-AdamW/pg2‚0.00013967327323848662 trainer/global_step‚13799 $ _timestamp‚1.784519338535159e+09 _runtime‚12512.112045973 _step‚1134î&%˜ 4,/ _wandbruntime‚12512  _runtime‚12512t-Á™ (‘,†P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111722777 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136532052 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2115905898662023 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2099901461488432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136542195 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2112263396404736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136532052 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _runtime‚12512.112113303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2128292373078528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2099901478527792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458783650210304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111722775 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248254827 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _step‚1135 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32931718 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16028618210304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2128292373078528 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16004777734865 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16028959634432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2112263413444096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2112263396404736 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57006188 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2128292014615040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24074760 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2128292014615040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16028959634432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2115906256262657 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57006478 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32931431 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16118807846912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111722777 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚474902458057216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24074757 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16118706771968 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2112263413444096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248264972 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336ÙÌ´×’,‹P ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2099901461488432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136542195 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2112263396404736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136532052 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _runtime‚12512.112113303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2128292373078528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2099901478527792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458783650210304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111722775 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248254827 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _step‚1135 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32931718 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16028618210304 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2128292373078528 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16004777734865 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16028959634432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2112263413444096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2112263396404736 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57006188 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2128292014615040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24074760 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2128292014615040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16028959634432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2115906256262657 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57006478 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32931431 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16118807846912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111722777 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚474902458057216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24074757 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16118706771968 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2112263413444096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248264972 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111722777 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136532052 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2115905898662023ïÒ7ª4“,/ _wandbruntime‚12512  _runtime‚12512t/ÐÑ'”,N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2100048892155904 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16030069740544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2128441399216128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136541708 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111730568 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136551865 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2128441757686784 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2112411687946240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2112411670906880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458816811868160 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2116054795318493 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16005886123229 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16029728309248 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2128441757686784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32933698 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚1136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248282435 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16005545561843 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 248272276 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136541708 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16119922295808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚13799 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2112411687946240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32933981 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚474936626804736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 248272276 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16029728309248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111730568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136551865 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57010475 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57010189 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24076494 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2116054437717747 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845193391443172e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16119814936576͉ç´'•,•N QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 248272276 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136541708 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16119922295808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚13799 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2112411687946240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32933981 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚474936626804736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 248272276 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16029728309248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111730568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136551865 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57010475 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57010189 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24076494 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2116054437717747 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845193391443172e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16119814936576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2100048892155904 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16030069740544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2128441399216128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136541708 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111730568 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136551865 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2128441757686784 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2112411687946240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2112411670906880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458816811868160 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2116054795318493 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16005886123229 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16029728309248 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2128441757686784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32933698 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚1136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248282435 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16005545561843ðÛ4–,/ _wandbruntime‚12512  _runtime‚12512ÎÛ‹—,‰ _step‚1137 0loss/FRt2/assignment_loss‚0.6180792123579578 1loss/SRqqt2/detection_loss‚0.6160465397070058 trainer/global_step‚13799 /loss/FBt2/detection_loss‚0.3064681760604597 % _timestamp‚1.7845193391473093e+09 _runtime‚12512.722642783 0loss/FRt1/assignment_loss‚1.5427621951738693 3loss/SRqqt1/assignment_loss‚0.37972552426736883 2loss/FBt1/assignment_loss‚0.014364734842010668 1loss/FBt2/assignment_loss‚0.08342087513266705 /loss/FRt2/detection_loss‚1.5149716860112206 1loss/SRqqt1/detection_loss‚0.5974489377892217 epoch‚34 2loss/SRqqt2/assignment_loss‚0.4460555595069002 0loss/FRt1/detection_loss‚0.45959044100404434 0loss/FBt1/detection_loss‚0.47368435645037255 %loss/total_loss‚7.052618238303099$¯´˜,Ž 0loss/FRt1/assignment_loss‚1.5427621951738693 3loss/SRqqt1/assignment_loss‚0.37972552426736883 2loss/FBt1/assignment_loss‚0.014364734842010668 1loss/FBt2/assignment_loss‚0.08342087513266705 /loss/FRt2/detection_loss‚1.5149716860112206 1loss/SRqqt1/detection_loss‚0.5974489377892217 epoch‚34 2loss/SRqqt2/assignment_loss‚0.4460555595069002 0loss/FRt1/detection_loss‚0.45959044100404434 0loss/FBt1/detection_loss‚0.47368435645037255 %loss/total_loss‚7.052618238303099 _step‚1137 0loss/FRt2/assignment_loss‚0.6180792123579578 1loss/SRqqt2/detection_loss‚0.6160465397070058 trainer/global_step‚13799 /loss/FBt2/detection_loss‚0.3064681760604597 % _timestamp‚1.7845193391473093e+09 _runtime‚12512.722642783ñŽ3x¹‚™,‚(:÷ ·µöÒºáÌË disk.nvme1n1.in‚ 229.87109375" disk.dm-1.out‚14468.3447265625 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.32421875' disk.nvme0c0n1.out‚83660.3212890625$ memory_percent‚97.01030254364014# proc.memory.rssMB‚ 3024.74609375 disk.md0.out‚ 632.078125$ disk.dm-0.out‚107590.05419921875( proc.memory.availableMB‚ 489.83203125) proc.memory.percent‚18.46158504486084 cpu‚25.687180234880554 proc.cpu.threads‚17 network.sent‚ 2.2130019e+07 network.recv‚ 3.264806e+06" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.1953125) disk./.usagePercent‚76.19007249119964% disk./.usageGB‚1169.7080879211426 disk.nvme0n1.in‚ 309.32421875% disk.nvme0n1.out‚83660.3212890625( disk.nvme1c1n1.out‚39714.19775390625& disk.nvme1n1.out‚39714.19775390625 disk.md0.in‚0 disk.dm-1.in‚0O3Ý)Ÿš,‚(:” ·µöÒ–•Ì gpu.0.gpu‚57.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.085' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.361666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&F$‹|›,‚(:ñ ƵöÒ›âË network.recv‚ 3.269233e+06 disk.nvme0n1.in‚ 309.32421875# disk.nvme0n1.out‚83835.57421875 disk.dm-1.in‚0" disk.dm-1.out‚14468.4423828125) proc.memory.percent‚18.46158504486084 cpu‚25.6869160776643 network.sent‚ 2.2193126e+07" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 632.4453125$ disk.dm-0.out‚107764.96435546875% disk.nvme0c0n1.out‚83835.57421875( proc.memory.availableMB‚ 489.97265625# proc.memory.rssMB‚ 3024.74609375( disk./.usagePercent‚76.1900230448713% disk./.usageGB‚1169.7073287963867& disk.nvme1n1.out‚39714.73974609375" disk.nvme0c0n1.in‚ 309.32421875$ memory_percent‚97.00944423675537 proc.cpu.threads‚17( disk.nvme1c1n1.out‚39714.73974609375 disk.md0.in‚0 disk.dm-0.in‚ 539.1953125 disk.rbd0.in‚0 disk.rbd0.out‚0¼¥çEžœ,‚(:“ ƵöÒ†Áìå gpu.0.gpu‚78.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚286.207' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.40233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qanž4,/ _wandbruntime‚12554  _runtime‚12554=Ò[Ìž,Æ ' lr-AdamW/pg1‚0.00013761480262883196 ' lr-AdamW/pg2‚0.00013761480262883196 trainer/global_step‚13849 % _timestamp‚1.7845193809460607e+09 _runtime‚12554.523051097 _step‚1138â;ŽÑŸ,Ë trainer/global_step‚13849 % _timestamp‚1.7845193809460607e+09 _runtime‚12554.523051097 _step‚1138 ' lr-AdamW/pg1‚0.00013761480262883196 ' lr-AdamW/pg2‚0.00013761480262883196òÜÑ4 ,/ _wandbruntime‚12554  _runtime‚12554ì»&'”¡,†P E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112112425 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460442645102080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A7DeviceStatsMonitor.on_traÖ}x#in_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2123332915249031 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249139274 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33045504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚12554.523148187 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112112425 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57207270 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112112427 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16060260948273 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16059920386999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2119677121511936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚460442663714304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845193809466891e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2135761308824064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33045810 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚306 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137026847 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476617305658880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16084187312128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16174534578176 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2123333272849665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249129129 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16174641944576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112112427 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137016704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57206961 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2135761308824064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476617179680256 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2107272994862032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137016704 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2107273011901392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚13849 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16084187312128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16084528736256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24161457 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16084528736256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _step‚1139 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2119677138551296 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2119677138551296 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2135761667287552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2119677121511936 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2135761667287552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24161460 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0ÈW(¢,‹P VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249139274 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33045504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚12554.523148187 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112112425 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57207270 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112112427 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16060260948273 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16059920386999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2119677121511936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚460442663714304 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 % _timestamp‚1.7845193809466891e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2135761308824064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33045810 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚306 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137026847 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚476617305658880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16084187312128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16174534578176 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2123333272849665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249129129 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16174641944576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112112427 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137016704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57206961 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2135761308824064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚476617179680256 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2107272994862032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137016704 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2107273011901392 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚13849 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16084187312128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16084528736256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24161457 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16084528736256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _step‚1139 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2119677138551296 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2119677138551296 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2135761667287552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2119677121511936 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2135761667287552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24161460 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112112425 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460442645102080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2123332915249031 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184óêµ ž|£,‚(:ñ ÕµöÒ‹³•Ì( disk.nvme1c1n1.out‚39715.79345703125 disk.md0.in‚0 disk.dm-0.in‚ 539.1953125# proc.memory.rssMB‚ 3024.74609375) disk./.usagePercent‚76.17455280442479 disk.nvme0n1.in‚ 309.32421875$ disk.nvme0n1.out‚83839.712890625" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚39715.79345703125$ disk.dm-0.out‚107768.91357421875 disk.dm-1.in‚0 network.recv‚ 3.272492e+06 disk.md0.out‚632.9375 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.32421875& disk.nvme0c0n1.out‚83839.712890625' proc.memory.availableMB‚ 489.4140625) proc.memory.percent‚18.46158504486084% disk./.usageGB‚1169.4698219299316! disk.dm-1.out‚14468.642578125$ memory_percent‚97.01285362243652 cpu‚25.686651574225245 proc.cpu.threads‚17 network.sent‚ 2.2209034e+07ã(û£ž¤,‚(:“ ÕµöÒñ¸ÉË gpu.0.gpu‚92.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚271.293' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.43100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Sí¯4¥,/ _wandbruntime‚12555  _runtime‚12555ý´c–'¦,N TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112120220 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16085297411072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚460475825372160 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16061369336637 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112120220 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2107420442568864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249156737 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24163191 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137036517 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16085297411072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33048002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16061028775251 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2119825413053440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2135910693425152 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137036517 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2119825396014080 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚476651600439296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112120218 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16085638842368 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249156737 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16175756454912 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2123481454304755 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _runtime‚12555.133152745 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137026360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2119825413053440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112120218 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57210898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2119825396014080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2135911051895808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2135911051895808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 % _timestamp‚1.7845193815574071e+09¯¥¢>â§,•N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16061028775251 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2119825413053440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2135910693425152 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137036517 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2119825396014080 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚476651600439296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112120218 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16085638842368 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249156737 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16175756454912 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2123481454304755 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _runtime‚12555.133152745 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137026360 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2119825413053440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112120218 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57210898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2119825396014080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2135911051895808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2135911051895808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 % _timestamp‚1.7845193815574071e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112120220 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16085297411072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚460475825372160 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16061369336637 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112120220 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2107420442568864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249156737 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24163191 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137036517 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16085297411072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33048002 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2130759931836039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2143230603033088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚478332039759360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112502075 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16230494811136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137501356 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16140097838080 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137511499 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137511499 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚310 trainer/global_step‚13899 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2143230961496576 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2127090846619136 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16140097838080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16230381153280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137501356 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16139756413952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2127090863658496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250013576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16139756413952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112502075 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57408436 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚307 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33160279 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112502077 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2143230603033088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24248157 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250003431 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112502077 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2130760289436673 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0oÓ:Þ%´,ŠP \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16140097838080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16230381153280 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137501356 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16139756413952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2127090863658496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250013576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16139756413952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112502075 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57408436 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚307 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33160279 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112502077 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2143230603033088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24248157 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250003431 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112502077 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2130760289436673 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2127090846619136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2114644545274992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57408746 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24248160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2143230961496576 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33160586 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2130759931836039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2143230603033088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚478332039759360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112502075 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16230494811136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137501356 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16140097838080 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137511499 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137511499 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚310 trainer/global_step‚13899 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2143230961496576 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2127090846619136 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303÷/È„ù4µ,/ _wandbruntime‚12597  _runtime‚12597ÜËå'¶,N OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112509868 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250031039 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚462134838876160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2143379987634176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24249894 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2130908828492509 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2143380346104832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33162523 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57412414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2114791975942464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250020880 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16231605229568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24249891 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112509868 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚462134857488384 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2130908470891763 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16116511988659 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112509870 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137511012 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250031039 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚478366334649344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2114791958903104 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚12597.11833671 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2127239138160640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16231495773184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2127239121121280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250020880 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137521169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16140866512896 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2143380346104832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57412714 _step‚1144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16140866512896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112509870 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚478366462717952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16116852550045 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137511012 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137521169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2127239138160640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296û]ñ'·,”N C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2114791975942464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250020880 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16231605229568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24249891 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112509868 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚462134857488384 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2130908470891763 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16116511988659 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112509870 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137511012 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250031039 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚478366334649344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2114791958903104 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚12597.11833671 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2127239138160640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16231495773184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2127239121121280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚300 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250020880 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137521169 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16140866512896 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2143380346104832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57412714 _step‚1144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16140866512896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112509870 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚478366462717952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16116852550045 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137511012 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137521169 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2127239138160640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112509868 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250031039 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚462134838876160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2143379987634176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24249894 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2130908828492509 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2143380346104832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33162523 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57412414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ø¯záJ4¸,/ _wandbruntime‚12597  _runtime‚12597IËW`¹,ˆ 0loss/FRt1/assignment_loss‚1.5544553016035976 0loss/FRt2/assignment_loss‚0.5997017278682952 /loss/FRt1/detection_loss‚0.4728673058884506 /loss/FBt1/detection_loss‚0.4004452296617003 % _timestamp‚1.7845194235453765e+09 2loss/SRqqt2/assignment_loss‚0.4941698558813072 /loss/FRt2/detection_loss‚1.4725902399799202 /loss/FBt2/detection_loss‚0.5392098563412825 trainer/global_step‚13899 _step‚1145 2loss/SRqqt1/assignment_loss‚0.4714116535389803 3loss/FBt1/assignment_loss‚0.0006502552444507118 0loss/SRqqt2/detection_loss‚0.615082007183928 epoch‚34 3loss/FBt2/assignment_loss‚6.701344351809111e-05 1loss/SRqqt1/detection_loss‚0.5767996466974727 %loss/total_loss‚7.197450093332904 _runtime‚12597.120785024d8}šº, 3loss/FBt1/assignment_loss‚0.0006502552444507118 0loss/SRqqt2/detection_loss‚0.615082007183928 epoch‚34 3loss/FBt2/assignment_loss‚6.701344351809111e-05 1loss/SRqqt1/detection_loss‚0.5767996466974727 %loss/total_loss‚7.197450093332904 _runtime‚12597.120785024 0loss/FRt1/assignment_loss‚1.5544553016035976 0loss/FRt2/assignment_loss‚0.5997017278682952 /loss/FRt1/detection_loss‚0.4728673058884506 /loss/FBt1/detection_loss‚0.4004452296617003 % _timestamp‚1.7845194235453765e+09 2loss/SRqqt2/assignment_loss‚0.4941698558813072 /loss/FRt2/detection_loss‚1.4725902399799202 /loss/FBt2/detection_loss‚0.5392098563412825 trainer/global_step‚13899 _step‚1145 2loss/SRqqt1/assignment_loss‚0.4714116535389803ùdsäîp»,‚(:å ‚¶öÒ¬çË network.recv‚ 3.284314e+06) disk./.usagePercent‚76.17449242523993$ disk./.usageGB‚1169.468894958496# disk.dm-1.out‚14469.01708984375 disk.nvme1n1.out‚ 39719.625" disk.nvme0c0n1.in‚ 309.32421875 disk.md0.in‚0 disk.dm-0.in‚ 539.1953125$ disk.dm-0.out‚107781.53662109375$ memory_percent‚97.01178073883057) proc.memory.percent‚18.46158504486084 disk.nvme1c1n1.out‚ 39719.625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.32421875 disk.md0.out‚ 634.41796875 cpu‚25.685947719015854 proc.cpu.threads‚17 disk.rbd0.in‚0% disk.nvme0c0n1.out‚83852.06640625 disk.nvme1n1.in‚ 229.87109375( proc.memory.availableMB‚ 489.58984375# proc.memory.rssMB‚ 3024.74609375# disk.nvme0n1.out‚83852.06640625" disk.nvme1c1n1.in‚ 229.87109375 network.sent‚ 2.2317526e+07]ž¼,‚(:“ ‚¶öÒ¾î­Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.559' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.18633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!ZtÀ½,‚(:õ ‘¶öÒ¶ÆÆË disk.dm-0.out‚107783.5703125 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.32421875( disk.nvme0c0n1.out‚83854.45166015625" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚39720.56201171875) disk./.usagePercent‚76.18074676459008 disk.md0.in‚0" disk.dm-1.out‚14469.1884765625& disk.nvme1n1.out‚39720.56201171875' proc.memory.availableMB‚ 489.5859375) proc.memory.percent‚18.46158504486084 cpu‚25.685688400961883 network.sent‚ 2.2369106e+07 disk.dm-0.in‚ 539.1953125& disk.nvme0n1.out‚83854.45166015625$ memory_percent‚97.01180458068848 network.recv‚ 3.288341e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.32421875 disk.nvme1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17% disk./.usageGB‚1169.5649147033691 disk.md0.out‚ 634.9453125Á e£“¾,‚(:ˆ ‘¶öÒú‘ŽÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚122.124' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úèð8¿,‚(:ô  ¶öÒꦽË( disk.nvme1c1n1.out‚39748.90966796875$ disk.dm-0.out‚107955.86376953125% disk./.usageGB‚1169.7098655700684" disk.dm-1.out‚14469.2939453125 disk.rbd0.out‚0 disk.md0.out‚ 635.1796875 disk.dm-0.in‚ 539.1953125$ memory_percent‚97.01175689697266# proc.memory.rssMB‚ 3024.74609375' disk.nvme0c0n1.out‚83999.0185546875 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚39748.90966796875 disk.md0.in‚0% proc.memory.availableMB‚ 489.59375 cpu‚25.68548823420501 disk.dm-1.in‚0% disk.nvme0n1.out‚83999.0185546875" disk.nvme1c1n1.in‚ 229.87109375) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17 network.sent‚ 2.2373939e+07 network.recv‚ 3.291732e+06) disk./.usagePercent‚76.19018828008913 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.32421875 disk.nvme0n1.in‚ 309.32421875TX*#žÀ,‚(:“  ¶öÒ²®„Ì gpu.0.gpu‚92.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚263.898' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.96600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ѤË4Á,/ _wandbruntime‚12638  _runtime‚126385ÊÒËÂ,Å ' lr-AdamW/pg1‚0.00013353067795757165 ' lr-AdamW/pg2‚0.00013353067795757165 trainer/global_step‚13949 $ _timestamp‚1.784519465310746e+09 _runtime‚12638.888039493 _step‚1146 2UØÐÃ,Ê ' lr-AdamW/pg1‚0.00013353067795757165 ' lr-AdamW/pg2‚0.00013353067795757165 trainer/global_step‚13949 $ _timestamp‚1.784519465310746e+09 _runtime‚12638.888039493 _step‚1146úŒ,ƒŸ4Ä,/ _wandbruntime‚12638  _runtime‚12638µü®WÅ,†P VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250877733 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33274499 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚13949 _runtime‚12638.888102653 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33274788 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16195666939904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2138187306023681 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2134504571726336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137986008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16195666939904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845194653113375e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2150700255705600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16195325515776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250887878 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2150699897242112 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2134504588765696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1147 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2134504588765696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16170886813815 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112891725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480046914591232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137996151 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2134504571726336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2134504571726336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137986008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16195666939904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845194653113375e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2150700255705600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16195325515776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250887878 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2150699897242112 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2134504588765696 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1147 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2134504588765696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16170886813815 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112891725 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480046914591232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488ûplò4Ç,/ _wandbruntime‚12639  _runtime‚12639éXe­'È,N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16287470798848 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2134652846228480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112899520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16287352953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2150849281843200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 % _timestamp‚1.7845194659225445e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57613609 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250895182 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16196777046016 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2134652863267840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24336594 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2138335487478771 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137995664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16171995202067 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2122163492276704 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137995664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚463793870992384 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138005821 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚13949 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16196777046016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57613314 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2138335845079517 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2150849640313856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33277015 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16172335763453 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24336591 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚1148 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250905341 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250905341 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33276723 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480081205334016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250895182 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112899518 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2150849640313856 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480081341791232 _runtime‚12639.498276252 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16196435614720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3Ïw&ÖÍÉ,•N C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2138335845079517 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2150849640313856 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33277015 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16172335763453 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24336591 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚1148 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250905341 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250905341 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33276723 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480081205334016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250895182 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112899518 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2150849640313856 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480081341791232 _runtime‚12639.498276252 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16196435614720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16287470798848 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2134652846228480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112899520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16287352953856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2150849281843200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 % _timestamp‚1.7845194659225445e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57613609 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250895182 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16196777046016 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2134652863267840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24336594 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2138335487478771 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/se4d_Ngment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137995664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16171995202067 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2122163492276704 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137995664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚463793870992384 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138005821 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚13949 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16196777046016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57613314üð? A4Ê,/ _wandbruntime‚12639  _runtime‚12639Á… ªË,‰ 0loss/FRt1/assignment_loss‚1.5300059652003164 3loss/FBt1/assignment_loss‚0.0004408347810140185 /loss/FRt1/detection_loss‚0.4918782872636652 /loss/FBt1/detection_loss‚0.2673869271086296 _runtime‚12639.499074723 0loss/FRt2/assignment_loss‚0.7317689332870994 3loss/SRqqt1/assignment_loss‚0.38609304780373055 trainer/global_step‚13949 3loss/FBt2/assignment_loss‚0.0003181762087065772 1loss/SRqqt1/detection_loss‚0.6344494798733392 1loss/SRqqt2/detection_loss‚0.6072018932939793 /loss/FBt2/detection_loss‚0.2663689550404966 _step‚1149 2loss/SRqqt2/assignment_loss‚0.5214673036511913 /loss/FRt2/detection_loss‚1.4801105705827813 $loss/total_loss‚6.91749037409495 epoch‚34 % _timestamp‚1.7845194659238276e+09›Ï’–Ì,Ž trainer/global_step‚13949 3loss/FBt2/assignment_loss‚0.0003181762087065772 1loss/SRqqt1/detection_loss‚0.6344494798733392 1loss/SRqqt2/detection_loss‚0.6072018932939793 /loss/FBt2/detection_loss‚0.2663689550404966 _step‚1149 2loss/SRqqt2/assignment_loss‚0.5214673036511913 /loss/FRt2/detection_loss‚1.4801105705827813 $loss/total_loss‚6.91749037409495 epoch‚34 % _timestamp‚1.7845194659238276e+09 0loss/FRt1/assignment_loss‚1.5300059652003164 3loss/FBt1/assignment_loss‚0.0004408347810140185 /loss/FRt1/detection_loss‚0.4918782872636652 /loss/FBt1/detection_loss‚0.2673869271086296 _runtime‚12639.499074723 0loss/FRt2/assignment_loss‚0.7317689332870994 3loss/SRqqt1/assignment_loss‚0.38609304780373055ýþŒã¶zÍ,‚(:ï ¯¶öÒ®âåË disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚39750.9072265625 disk.nvme0n1.in‚ 309.32421875 network.sent‚ 2.2389704e+07 network.recv‚ 3.29502e+06 disk.md0.in‚0 disk.md0.out‚ 635.67578125 disk.dm-0.in‚ 539.1953125" disk.nvme0c0n1.in‚ 309.32421875& disk.nvme0c0n1.out‚84024.404296875$ disk.nvme0n1.out‚84024.404296875$ disk./.usageGB‚1169.706916809082 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚39750.9072265625# memory_percent‚97.0134973526001 proc.cpu.threads‚17# disk.dm-0.out‚107981.9951171875! disk.dm-1.out‚14469.474609375( proc.memory.availableMB‚ 489.30859375# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 cpu‚25.68528687153178) disk./.usagePercent‚76.18999620967803WÓAzžÎ,‚(:“ ¯¶öÒ졦Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚287.698' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.89933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç>á/~Ï,‚(:ó ¾¶öÒ¬ÐÊË$ disk./.usageGB‚1169.469524383545& disk.nvme0n1.out‚84080.06884765625 disk.rbd0.in‚0# proc.memory.rssMB‚ 3024.74609375 cpu‚25.685048804809952 proc.cpu.threads‚17 network.recv‚ 3.299264e+06 disk.nvme0n1.in‚ 309.32421875" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚39751.41552734375& disk.nvme1n1.out‚39751.41552734375 disk.md0.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚84080.06884765625) disk./.usagePercent‚76.17453342345188 disk.nvme1n1.in‚ 229.87109375! disk.dm-1.out‚14469.556640625$ memory_percent‚97.01213836669922% proc.memory.availableMB‚ 489.53125) proc.memory.percent‚18.46158504486084 network.sent‚ 2.2441043e+07 disk.md0.out‚ 636.015625 disk.dm-0.in‚ 539.1953125# disk.dm-0.out‚108037.3505859375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.32421875©¾zrœÐ,‚(:‘ ¾¶öÒø±‘Ì gpu.0.gpu‚25.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.39' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.46333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 yÑ,‚(:î ͶöÒäÙÊË% proc.memory.availableMB‚ 489.53125) proc.memory.percent‚18.46158504486084) disk./.usagePercent‚76.17452646617954( disk.nvme1c1n1.out‚39754.86669921875 disk.md0.in‚0 disk.dm-1.in‚0# disk.nvme0n1.out‚84096.60546875 cpu‚25.68484979672799 proc.cpu.threads‚17& disk.nvme1n1.out‚39754.86669921875 disk.md0.out‚ 636.453125# disk.dm-1.out‚14469.68603515625 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.32421875# proc.memory.rssMB‚ 3024.74609375 network.recv‚ 3.302646e+06" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.1953125% disk.nvme0c0n1.out‚84096.60546875$ memory_percent‚97.01213836669922 network.sent‚ 2.244598e+07% disk./.usageGB‚1169.4694175720215 disk.nvme1n1.in‚ 229.87109375# disk.dm-0.out‚108056.2685546875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.324218751‡óžÒ,‚(:“ ͶöÒß³“Ì gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚122.008' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.66933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚.ºÙ4Ó,/ _wandbruntime‚12680  _runtime‚12680T¤äÊÔ,Ä & lr-AdamW/pg2‚0.0001315052862116615 trainer/global_step‚13999 % _timestamp‚1.7845195072495077e+09 _runtime‚12680.826998942 _step‚1150 & lr-AdamW/pg1‚0.0001315052862116615±ï«¾ÏÕ,É & lr-AdamW/pg1‚0.0001315052862116615 & lr-AdamW/pg2‚0.0001315052862116615 trainer/global_step‚13999 % _timestamp‚1.7845195072495077e+09 _runtime‚12680.826998942 _step‚1150þß4Ö,/ _wandbruntime‚12680  _runtime‚12680-ó‚ý (×,†P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16251236041728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138470660 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33388853 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2145613965010055 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138470660 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2158169549914624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33389135 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465419704226304 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 % _timestamp‚1.7845195072500844e+09 _runtime‚12680.827112583 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24421557 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16342189975040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138480803 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16250894617600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2141918296833536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16226370027223 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113281375 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113281377 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16251236041728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113281375 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚481761772417024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚465419685614080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2158169191451136 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2141918296833536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚13999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2158169549914624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚1151 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138480803 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16250894617600 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113281377 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2129387594982832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251762180 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2141918313872896^A¬(Ø,‹P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚481761894201344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24421560 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16342086802944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251752035 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2129387612022192 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16226710588497 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2158169191451136 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2141918296833536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚13999 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2158169549914624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚1151 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138480803 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16250894617600 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113281377 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2129387594982832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251762180 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2141918313872896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16251236041728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138470660 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33388853 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2145613965010055 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138470660 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2158169549914624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33389135 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465419704226304 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 % _timestamp‚1.7845195072500844e+09 _runtime‚12680.827112583 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24421557 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16342189975040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138480803 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16250894617600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2141918296833536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16226370027223 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113281375 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113281377 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16251236041728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113281375 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚481761772417024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚465419685614080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24423294 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚481796067626496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251769484 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2158318576052224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33391393 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16227478415475 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113289168 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2145762861666525 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2142066571335680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16343201742336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251779643 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113289170 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2142066571335680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57814379 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _step‚1152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251779643 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113289170 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16343309101568 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2129535042689664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16252004716544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24423294 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚481796067626496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251769484 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2158318576052224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33391393 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16227478415475 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113289168 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2145762861666525 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2142066571335680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16343201742336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251779643 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113289170 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2142066571335680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57814379 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _step‚1152 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251779643 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113289170 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16343309101568 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2129535042689664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16252004716544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799€ mÎÍ4Ü,/ _wandbruntime‚12681  _runtime‚12681K´‡d Ý,… 1loss/FBt2/assignment_loss‚0.06671391843283792 /loss/FBt2/detection_loss‚0.6801506528697375 1loss/SRqqt2/detection_loss‚0.5987365566424664 0loss/FBt1/detection_loss‚0.36293331715838845 %loss/total_loss‚7.285925135141044 trainer/global_step‚13999 $ _timestamp‚1.784519508167165e+09 _step‚1153 /loss/FRt1/detection_loss‚0.4389516213883322 1loss/SRqqt1/detection_loss‚0.6326810490135558 epoch‚34 0loss/FRt2/assignment_loss‚0.6619977027458964 3loss/SRqqt1/assignment_loss‚0.44041392741895746 2loss/SRqqt2/assignment_loss‚0.4808039770036097 3loss/FBt1/assignment_loss‚0.0001614002195851801 .loss/FRt2/detection_loss‚1.417266918448424 _runtime‚12681.74250364 /loss/FRt1/assignment_loss‚1.505114093799252PÍ9Þ,Š 0loss/FRt2/assignment_loss‚0.6619977027458964 3loss/SRqqt1/assignment_loss‚0.44041392741895746 2loss/SRqqt2/assignment_loss‚0.4808039770036097 3loss/FBt1/assignment_loss‚0.0001614002195851801 .loss/FRt2/detection_loss‚1.417266918448424 _runtime‚12681.74250364 /loss/FRt1/assignment_loss‚1.505114093799252 1loss/FBt2/assignment_loss‚0.06671391843283792 /loss/FBt2/detection_loss‚0.6801506528697375 1loss/SRqqt2/detection_loss‚0.5987365566424664 0loss/FBt1/detection_loss‚0.36293331715838845 %loss/total_loss‚7.285925135141044 trainer/global_step‚13999 $ _timestamp‚1.784519508167165e+09 _step‚1153 /loss/FRt1/detection_loss‚0.4389516213883322 1loss/SRqqt1/detection_loss‚0.6326810490135558 epoch‚34 <[~ß,‚(:ö ܶöÒóÔ€Ì( disk.nvme0c0n1.out‚84097.51513671875# proc.memory.rssMB‚ 3023.42578125 cpu‚25.6863218690767 proc.cpu.threads‚17% disk./.usageGB‚1169.4696502685547" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚39755.2060546875 disk.md0.in‚0$ memory_percent‚97.00541496276855' proc.memory.availableMB‚ 490.6328125 network.sent‚ 2.2461787e+07) disk./.usagePercent‚76.17454162309426" disk.nvme0c0n1.in‚ 309.32421875& disk.nvme0n1.out‚84097.51513671875 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.1953125# disk.dm-0.out‚108057.0419921875# disk.dm-1.out‚14469.74853515625 network.recv‚ 3.30603e+06 disk.nvme0n1.in‚ 309.32421875% disk.nvme1n1.out‚39755.2060546875 disk.md0.out‚ 636.64453125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚18.453526496887207(í-žà,‚(:“ ܶöÒçîÊÌ gpu.0.gpu‚64.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚284.545' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.84833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’‹Q4á,/ _wandbruntime‚12697  _runtime‚12697´qR°06â,ýy 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7021359205245972 7"Purity/*FRt*SRqqt0FBt/event_purity‚0.28265380859375 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.894271194934845 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2736099362373352 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3838845193386078 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6069132089614868 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2708350718021393 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8921012878417969 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6475051045417786 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5633591413497925 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6322658061981201 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6620334982872009 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_2‚0.1499100923538208 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5807915925979614 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7006312608718872 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.282719224691391 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.049311596900224686 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5054904222488403 (particle/accuracy‚0.8500860333442688 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10486247390508652 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.683125376701355 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2849664092063904 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26213207840919495 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9171246886253357 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.651665210723877 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3250223994255066 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5498315095901489 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5677083134651184 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 particle/accuracy_1_of_1‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5602828860282898 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_2‚0.7430798411369324 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1706084907054901 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 epoch‚34 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.726211667060852 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6582711338996887 jet/accuracy_2_of_5‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2704892158508301 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6867883205413818 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10631082206964493 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6937930583953857 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 $ _timestamp‚1.784519523928739e+09 )loss/val_total_loss‚7.474619498920746 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 *particle/sensitivity‚0.636047899723053 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2643142342567444 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28261667490005493 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6204166412353516 particle/accuracy_5_of_5‚0 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.535202145576477 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6087037324905396 particle/accuracy_1_of_2‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.57600337266922 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9591003656387329 _runtime‚12697.504781649 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5830086469650269 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 jet/accuracy_1_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6268572211265564 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10156086832284927 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7021359205245972 trainer/global_step‚13999 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21532826125621796 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03673936799168587 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3321317136287689 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7validation_average_jet_accuracy‚0.35381418466567993 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9570599794387817 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33169957995414734 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22806903719902039 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2866228520870209 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2523616552352905 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22063662111759186 particle/accuracy_3_of_4‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6442934274673462 _step‚1154 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 jet/accuracy_4_of_5‚0 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.546969473361969 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6075223088264465 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05266941711306572 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07897676527500153 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8896258473396301 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5090653896331787 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9070885181427002 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5126526951789856 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8934807777404785 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7682291865348816 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27183085680007935 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16737177968025208 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5964854955673218 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 'particle/f_score‚0.6456714272499084 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10564899444580078 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04046109318733215 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3294452130794525 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8736096620559692 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8136621117591858 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2704892158508301 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5460083484649658 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3132619857788086 +jet/accuracy_3_of_3‚0.08193347603082657 particle/accuracy_1_of_4‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6542845964431763 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2041362076997757 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 /particle/accuracy_4_of_4‚0.9514349699020386 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5932212471961975 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5460083484649658 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2527301609516144 +particle/specificity‚0.9086297154426575 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.766940176486969 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.474496066570282 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5815072059631348 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2433333396911621 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28917360305786133 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32104042172431946 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23395875096321106 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26498693227767944 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.372616171836853 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24185723066329956 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3095104694366455 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5885325074195862 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6584131121635437 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6187697052955627 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9910994172096252 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9016251564025879 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10156086832284927 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9972196221351624 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952044486999512 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5994219779968262 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8955782055854797 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5649476051330566 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988906383514404 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10261107981204987 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9968922734260559 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6221722364425659 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 particle/accuracy_4_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.51292783021926¿šêªÓ88 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9913375377655029 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 +jet/accuracy_4_of_4‚0.01670061983168125 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992322325706482 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.528916597366333 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25236812233924866 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.10696525871753693 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6515800952911377 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6759454011917114 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 *jet/accuracy_2_of_4‚0.6910400986671448 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1050453633069992 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5615234971046448 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645D+Ø¥=ã,‚z 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10631082206964493 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6937930583953857 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 $ _timestamp‚1.784519523928739e+09 )loss/val_total_loss‚7.474619498920746 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 *particle/sensitivity‚0.636047899723053 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2643142342567444 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.28261667490005493 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6204166412353516 particle/accuracy_5_of_5‚0 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.535202145576477 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6087037324905396 particle/accuracy_1_of_2‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.57600337266922 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9591003656387329 _runtime‚12697.504781649 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5830086469650269 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 jet/accuracy_1_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6268572211265564 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10156086832284927 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7021359205245972 trainer/global_step‚13999 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.21532826125621796 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03673936799168587 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3321317136287689 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7validation_average_jet_accuracy‚0.35381418466567993 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9570599794387817 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33169957995414734 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22806903719902039 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2866228520870209 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2523616552352905 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22063662111759186 particle/accuracy_3_of_4‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6442934274673462 _step‚1154 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 jet/accuracy_4_of_5‚0 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.546969473361969 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6075223088264465 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05266941711306572 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07897676527500153 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8896258473396301 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5090653896331787 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9070885181427002 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5126526951789856 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8934807777404785 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7682291865348816 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.27183085680007935 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16737177968025208 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5964854955673218 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 'particle/f_score‚0.6456714272499084 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10564899444580078 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04046109318733215 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3294452130794525 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8736096620559692 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8136621117591858 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2704892158508301 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5460083484649658 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.3132619857788086 +jet/accuracy_3_of_3‚0.08193347603082657 particle/accuracy_1_of_4‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6542845964431763 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2041362076997757 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 /particle/accuracy_4_of_4‚0.9514349699020386 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5932212471961975 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5460083484649658 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2527301609516144 +particle/specificity‚0.9086297154426575 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.766940176486969 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.474496066570282 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5815072059631348 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2433333396911621 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28917360305786133 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32104042172431946 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23395875096321106 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26498693227767944 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.372616171836853 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24185723066329956 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3095104694366455 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5885325074195862 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6584131121635437 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6187697052955627 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9910994172096252 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.9016251564025879 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10156086832284927 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9972196221351624 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952044486999512 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5994219779968262 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8955782055854797 particle/accuracy_1_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5649476051330566 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988906383514404 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10261107981204987 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9968922734260559 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6221722364425659 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 particle/accuracy_4_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9913375377655029 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 +jet/accuracy_4_of_4‚0.01670061983168125 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992322325706482 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.528916597366333 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25236812233924866 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.10696525871753693 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6515800952911377 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6759454011917114 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 *jet/accuracy_2_of_4‚0.6910400986671448 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1050453633069992 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5615234971046448 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7021359205245972 7"Purity/*FRt*SRqqt0FBt/event_purity‚0.28265380859375 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.894271194934845 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2736099362373352 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3838845193386078 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6069132089614868 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2708350718021393 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8921012878417969 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6475051045417786 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5633591413497925 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6322658061981201 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6620334982872009 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_2‚0.1499100923538208 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5807915925979614 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7006312608718872 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.282719224691391 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.049311596900224686 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5054904222488403 (particle/accuracy‚0.8500860333442688 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10486247390508652 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.683125376701355 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2849664092063904 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.26213207840919495 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9171246886253357 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.651665210723877 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3250223994255066 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5498315095901489 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5677083134651184 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 particle/accuracy_1_of_1‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5602828860282898 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_2‚0.7430798411369324 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1706084907054901 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 epoch‚34 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.726211667060852 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6582711338996887 jet/accuracy_2_of_5‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2704892158508301 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6867883205413818 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129‚ „0›|ä,‚(:ñ ë¶öÒ¬úïË disk.md0.in‚0# proc.memory.rssMB‚ 3024.74609375 cpu‚25.68779280437163 proc.cpu.threads‚17) disk./.usagePercent‚76.17872045902413" disk.nvme0c0n1.in‚ 309.32421875' disk.nvme0c0n1.out‚84158.0712890625$ memory_percent‚97.01271057128906 network.sent‚ 2.2524065e+07 disk.md0.out‚ 637.2734375 disk.dm-0.in‚ 539.1953125$ disk.dm-0.out‚108117.05615234375 disk.dm-1.in‚0 disk.rbd0.out‚0) proc.memory.percent‚18.46158504486084 network.recv‚ 3.310128e+06 disk.nvme0n1.in‚ 309.32421875% disk.nvme0n1.out‚84158.0712890625 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚39756.1865234375# disk.dm-1.out‚14469.96044921875 disk.rbd0.in‚0$ proc.memory.availableMB‚489.4375$ disk./.usageGB‚1169.533805847168" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚39756.1865234375Ì’Àžå,‚(:“ ë¶öÒÝœ¾Ì gpu.0.gpu‚92.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚267.001' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.00033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–Æeªqæ,‚(:æ ú¶öÒœöÀË network.sent‚ 2.25699e+07 disk.md0.out‚ 637.515625& disk.nvme0c0n1.out‚84266.248046875! disk.nvme1c1n1.out‚ 39780.3125 disk.nvme1n1.in‚ 229.87109375$ memory_percent‚97.01268672943115 cpu‚25.687573193261674 proc.cpu.threads‚17) disk./.usagePercent‚76.17459231179267 disk.md0.in‚0# disk.dm-1.out‚14469.98388671875$ disk.nvme0n1.out‚84266.248046875" disk.nvme1c1n1.in‚ 229.87109375 network.recv‚ 3.314596e+06% disk./.usageGB‚1169.4704284667969 disk.dm-0.in‚ 539.1953125$ disk.dm-0.out‚108248.79443359375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.32421875 disk.nvme0n1.in‚ 309.32421875 disk.nvme1n1.out‚ 39780.3125( proc.memory.availableMB‚ 489.44140625# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084¾0.iç,‚(:’ ú¶öÒù’†Ì gpu.0.gpu‚24.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.998' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.99933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H¹Zþƒè,‚(:ø ‰·öÒýúØË& disk.nvme0n1.out‚84333.78173828125( proc.memory.availableMB‚ 489.44140625) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17 disk.md0.in‚0 disk.dm-0.in‚ 539.1953125 disk.nvme0n1.in‚ 309.32421875# proc.memory.rssMB‚ 3024.74609375 cpu‚25.687390953317966 network.sent‚ 2.2574729e+07 network.recv‚ 3.317782e+06) disk./.usagePercent‚76.15911163543765% disk./.usageGB‚1169.2327613830566" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚39782.2294921875" disk.dm-0.out‚108317.548828125 disk.dm-1.in‚0 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚39782.2294921875 disk.md0.out‚ 637.78515625 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.32421875( disk.nvme0c0n1.out‚84333.78173828125$ memory_percent‚97.01268672943115# disk.dm-1.out‚14470.08251953125ó« בé,‚(:† ‰·öÒ×®›Ì gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.714' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å¼Pð4ê,/ _wandbruntime‚12739  _runtime‚12739ê¨5gÌë,Æ trainer/global_step‚14049 % _timestamp‚1.7845195655259717e+09 _runtime‚12739.103018935 _step‚1155 ' lr-AdamW/pg1‚0.00012949118188060565 ' lr-AdamW/pg2‚0.00012949118188060565í~Ñì,Ë _step‚1155 ' lr-AdamW/pg1‚0.00012949118188060565 ' lr-AdamW/pg2‚0.00012949118188060565 trainer/global_step‚14049 % _timestamp‚1.7845195655259717e+09 _runtime‚12739.103018935ƒ ßl{ë4í,/ _wandbruntime‚12739  _runtime‚12739X( (î,†P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16327122979328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16418720338432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24512376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33538068 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚483519894946816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚14049 _step‚1156 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113793835 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2167601147371520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113793833 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113793833 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467101296392704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58050728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33538352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16302497650802 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2167601147371520 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16327122979328 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2151274382855680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139036806 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58050441 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113793835 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _runtime‚12739.103108405 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139046949Ö†&ˆvï,‹P TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16302497650802 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested;t™î›%_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2167601147371520 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16327122979328 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2151274382855680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139036806 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58050441 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113793835 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _runtime‚12739.103108405 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139046949 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16327122979328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16418720338432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24512376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33538068 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚483519894946816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚14049 _step‚1156 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113793835 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2167601147371520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113793833 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113793833 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467101296392704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58050728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33538352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16328233085440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113801628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚483554314333184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467134458050560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1157 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2151422640318464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2167750531972608 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2167750890443264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16327891654144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2151422640318464 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139056619 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚467134476662784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33540557 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16303265477780 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113801626 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24514110 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58054380 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252848088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252858247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 trainer/global_step‚14049 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24514107 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚ 12739.7124734 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452báoÑ'ò,“N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113801626 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16303606039166 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2151422657357824 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16327891654144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139046462 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚483554188361728 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2138827832071488 % _timestamp‚1.7845195661367388e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58054667 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16419837670400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113801628 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16328233085440 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113801628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚483554314333184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚467134458050560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1157 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2151422640318464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2167750531972608 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2167750890443264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16327891654144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2151422640318464 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139056619 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚467134476662784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33540557 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16303265477780 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113801626 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24514110 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58054380 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252848088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252858247 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 trainer/global_step‚14049 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24514107 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚ 12739.7124734 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2155131438110654 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2151422657357824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2… PÍØÈ4ó,/ _wandbruntime‚12739  _runtime‚12739bó/ƒô,ˆ 3loss/SRqqt2/assignment_loss‚0.46685412713462043 /loss/FRt1/detection_loss‚0.4715371496661172 /loss/FRt2/detection_loss‚1.4664849002270484 % _timestamp‚1.7845195661395504e+09 3loss/FBt1/assignment_loss‚4.400538576784693e-05 1loss/SRqqt1/detection_loss‚0.6938908020874449 1loss/SRqqt2/detection_loss‚0.5888662795811642 /loss/FBt1/detection_loss‚0.3437317997827281 /loss/FBt2/detection_loss‚0.3553559917339083 epoch‚35 trainer/global_step‚14049 _step‚1158 0loss/FRt2/assignment_loss‚0.6701614845732939 $loss/total_loss‚7.01320289944109 2loss/SRqqt1/assignment_loss‚0.4681757007330491 2loss/FBt2/assignment_loss‚0.009364164201821545 _runtime‚12739.714845515 0loss/FRt1/assignment_loss‚1.4787364943341266¢Nàõ, 3loss/SRqqt2/assignment_loss‚0.46685412713462043 /loss/FRt1/detection_loss‚0.4715371496661172 /loss/FRt2/detection_loss‚1.4664849002270484 % _timestamp‚1.7845195661395504e+09 3loss/FBt1/assignment_loss‚4.400538576784693e-05 1loss/SRqqt1/detection_loss‚0.6938908020874449 1loss/SRqqt2/detection_loss‚0.5888662795811642 /loss/FBt1/detection_loss‚0.3437317997827281 /loss/FBt2/detection_loss‚0.3553559917339083 epoch‚35 trainer/global_step‚14049 _step‚1158 0loss/FRt2/assignment_loss‚0.6701614845732939 $loss/total_loss‚7.01320289944109 2loss/SRqqt1/assignment_loss‚0.4681757007330491 2loss/FBt2/assignment_loss‚0.009364164201821545 _runtime‚12739.714845515 0loss/FRt1/assignment_loss‚1.4787364943341266† yåUÖ{ö,‚(:ð ˜·öÒ¹ËÙË disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.32421875) proc.memory.percent‚18.46158504486084 disk.rbd0.out‚0( disk.nvme0c0n1.out‚84454.53662109375$ memory_percent‚97.01290130615234# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17 network.sent‚ 2.2590595e+07 network.recv‚ 3.321386e+06 disk.md0.out‚ 638.2421875 disk.nvme0n1.in‚ 309.32421875" disk.nvme1c1n1.in‚ 229.87109375 cpu‚25.687131585383543 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-0.in‚ 539.1953125# disk.dm-1.out‚14470.23876953125 disk.rbd0.in‚0& disk.nvme0n1.out‚84454.53662109375% disk.nvme1c1n1.out‚39906.03515625% proc.memory.availableMB‚ 489.40625) disk./.usagePercent‚76.17461989240797% disk./.usageGB‚1169.4708518981934# disk.nvme1n1.out‚39906.03515625" disk.dm-0.out‚108560.982421875Åa'О÷,‚(:“ ˜·öÒñ«Ì gpu.0.gpu‚87.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚291.568' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.18933333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾¸öPø,‚(:í §·öÒú¹åË& disk.nvme0n1.out‚84467.94189453125# proc.memoryÏ '(.rssMB‚ 3024.74609375 cpu‚25.68687246764132 network.sent‚ 2.2641988e+07 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.1953125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚84467.94189453125 disk.nvme0n1.in‚ 309.32421875# disk.dm-1.out‚14490.36669921875' disk./.usagePercent‚76.174620389356" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.32421875% proc.memory.availableMB‚ 489.40625$ disk./.usageGB‚1169.470859527588" disk.dm-0.out‚108575.990234375 disk.rbd0.in‚0$ memory_percent‚97.01290130615234) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17 network.recv‚ 3.3253e+06& disk.nvme1c1n1.out‚39928.599609375$ disk.nvme1n1.out‚39928.599609375 disk.md0.in‚0 disk.md0.out‚ 638.62890625A@ýžù,‚(:“ §·öÒìͯÌ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚266.995' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.99833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=mÚú,‚(:ô ¶·öÒûå¨Ë network.sent‚ 2.264484e+07 disk.rbd0.out‚0& disk.nvme0n1.out‚84472.28857421875 disk.md0.in‚0' proc.memory.availableMB‚ 489.8046875 cpu‚25.686635867700332% disk./.usageGB‚1169.2332153320312( disk.nvme0c0n1.out‚84472.28857421875" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚40443.9130859375 disk.dm-0.in‚ 539.1953125# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 disk.nvme1n1.in‚ 229.87109375 network.recv‚ 3.328228e+06) disk./.usagePercent‚76.15914120384505 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.32421875 disk.nvme0n1.in‚ 309.32421875 disk.md0.out‚ 638.90234375 disk.dm-0.out‚108580.1171875% disk.nvme1n1.out‚40443.9130859375 disk.dm-1.in‚0# disk.dm-1.out‚15005.28857421875$ memory_percent‚97.01046943664551 proc.cpu.threads‚17sxòŸû,‚(:” ¶·öÒªùË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚148.596' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚49.532000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pþŠ”4ü,/ _wandbruntime‚12780  _runtime‚12780½áé.Ìý,Æ ' lr-AdamW/pg2‚0.00012748849432656176 trainer/global_step‚14099 % _timestamp‚1.7845196073774395e+09 _runtime‚12780.953022969 _step‚1159 ' lr-AdamW/pg1‚0.00012748849432656176k9ÚŠÑþ,Ë ' lr-AdamW/pg1‚0.00012748849432656176 ' lr-AdamW/pg2‚0.00012748849432656176 trainer/global_step‚14099 % _timestamp‚1.7845196073774395e+09 _runtime‚12780.953022969 _step‚1159‡ „ì¸4ÿ,/ _wandbruntime‚12780  _runtime‚12780ù¢‚¥ (€-†P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16382350657024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2175070441580544 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2158688107962880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚468760309896704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24599076 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2175070800044032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚485234889777664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2146051934777616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114183485 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16382692081152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 % _timestamp‚1.7845196073783011e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2162409915641826 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16382350657024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58252384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33653308 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139531601 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16474476708864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚468760291284480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2175070441580544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33653029 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 _runtime‚12780.956975307 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2162409915641826 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16382350657024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58252384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33653308 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139531601 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16474476708864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚468760291284480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2175070441580544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33653029 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 _runtime‚12780.956975307 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2175070441580544 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2158688107962880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚468760309896704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24599076 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2175070800044032 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚485234889777664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2146051934777616ˆ ñý§4‚-/ _wandbruntime‚12781  _runtime‚12781›zØír#ƒ-N ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16383802187264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114191278 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139531114 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253732549 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16358748691188 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚485269189755904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2175220184652288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16383460755968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚14099 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚468793490166784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2175219826181632 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58256108 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24600807 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚468793471554560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139531114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58256392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16475588035584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2158836382465024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2158836382465024 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚12781.558510901 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚485269059590144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114191276 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139541271 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7845196079826996e+09 _step‚1161 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253722390 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2146199348405728 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 253732549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 253722390 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114191276 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24600810 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16475699589120 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139541271 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2158836365425664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33655301 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2162558097096916 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528:´x'„-•N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚485269059590144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114191276 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139541271 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7845196079826996e+09 _step‚1161 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253722390 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2146199348405728 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 253732549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 253722390 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114191276 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24600810 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16475699589120 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139541271 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2158836365425664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33655301 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2162558097096916 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16383802187264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114191278 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139531114 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253732549 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16358748691188 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚485269189755904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2175220184652288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16383460755968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚14099 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚468793490166784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2175219826181632 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58256108 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24600807 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚468793471554560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139531114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58256392 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16475588035584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2158836382465024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2158836382465024 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚12781.558510901‰ &!Ö¼4…-/ _wandbruntime‚12781  _runtime‚12781µüc™†-Š 3loss/SRqqt2/assignment_loss‚0.44706497320001554 1loss/SRqqt1/detection_loss‚0.5583200211365633 1loss/SRqqt2/detection_loss‚0.6404799575030701 /loss/FBt2/detection_loss‚0.5612440682421236 trainer/global_step‚14099 0loss/FRt1/assignment_loss‚1.4602687936545107 /loss/FRt2/detection_loss‚1.4656659728635075 %loss/total_loss‚7.118487529226698 _runtime‚12781.566021997 _step‚1162 3loss/SRqqt1/assignment_loss‚0.36968020709946997 3loss/FBt1/assignment_loss‚0.0006241998674810276 /loss/FRt1/detection_loss‚0.4105690068766984 epoch‚35 $ _timestamp‚1.784519607986969e+09 0loss/FRt2/assignment_loss‚0.7308623668983261 2loss/FBt2/assignment_loss‚3.84679678183723e-05 0loss/FBt1/detection_loss‚0.47366949391711266¼…Û·- $ _timestamp‚1.784519607986969e+09 0loss/FRt2/assignment_loss‚0.7308623668983261 2loss/FBt2/assignment_loss‚3.84679678183723e-05 0loss/FBt1/detection_loss‚0.47366949391711266 3loss/SRqqt2/assignment_loss‚0.44706497320001554 1loss/SRqqt1/detection_loss‚0.5583200211365633 1loss/SRqqt2/detection_loss‚0.6404799575030701 /loss/FBt2/detection_loss‚0.5612440682421236 trainer/global_step‚14099 0loss/FRt1/assignment_loss‚1.4602687936545107 /loss/FRt2/detection_loss‚1.4656659728635075 %loss/total_loss‚7.118487529226698 _runtime‚12781.566021997 _step‚1162 3loss/SRqqt1/assignment_loss‚0.36968020709946997 3loss/FBt1/assignment_loss‚0.0006241998674810276 /loss/FRt1/detection_loss‚0.4105690068766984 epoch‚35Š Æžö€ˆ-‚(:õ Å·öҤߋË% disk./.usageGB‚1169.9841766357422" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚84938.04345703125( disk.nvme1c1n1.out‚40515.54736328125& disk.nvme1n1.out‚40515.54736328125( proc.memory.availableMB‚ 489.77734375) proc.memory.percent‚18.46158504486084 network.recv‚ 3.331558e+06 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875 disk.dm-1.out‚ 15005.4921875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚84938.04345703125" disk.nvme1c1n1.in‚ 229.87109375 network.sent‚ 2.2650084e+07 disk.dm-1.in‚0$ memory_percent‚97.01063632965088# proc.memory.rssMB‚ 3024.74609375 cpu‚25.686379267687638 proc.cpu.threads‚17) disk./.usagePercent‚76.20805579780115 disk.md0.out‚ 639.39453125! disk.dm-0.out‚109116.24609375 disk.rbd0.out‚0 disk.nvme1n1.in‚ 229.87109375£îM7Ÿ‰-‚(:” Å·öÒÿÝ€Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚128.739' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚42.913000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º\HÚzŠ-‚(:ï Ô·öҾ̹Ë( disk.nvme1c1n1.out‚40639.46630859375 disk.nvme1n1.in‚ 229.87109375 proc.cpu.threads‚17 network.recv‚ 3.335856e+06) disk./.usagePercent‚76.20561354673951 disk.dm-0.out‚ 109332.46875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚85031.06005859375" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚40639.46630859375$ memory_percent‚97.01066017150879% disk./.usageGB‚1169.9466819763184 disk.dm-1.out‚ 15005.515625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0n1.out‚85031.06005859375 disk.md0.out‚ 639.7109375' proc.memory.availableMB‚ 489.7734375# proc.memory.rssMB‚ 3024.74609375 network.sent‚ 2.2713189e+07 disk.dm-0.in‚ 539.2421875 disk.nvme0n1.in‚ 309.37109375 disk.md0.in‚0) proc.memory.percent‚18.46158504486084 cpu‚25.686179288744665 disk.dm-1.in‚0vlæÍŸ‹-‚(:” Ô·öÒÐýüË gpu.0.gpu‚94.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚151.979' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚50.659666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žc4Œ-/ _wandbruntime‚12822  _runtime‚12822ºAÓAÌ-Æ ' lr-AdamW/pg1‚0.00012549735217840885 ' lr-AdamW/pg2‚0.00012549735217840885 trainer/global_step‚14149 % _timestamp‚1.7845196489825995e+09 _runtime‚12822.558181272 _step‚1163YÌcuÑŽ-Ë ' lr-AdamW/pg1‚0.00012549735217840885 ' lr-AdamW/pg2‚0.00012549735217840885 trainer/global_step‚14149 % _timestamp‚1.7845196489825995e+09 _runtime‚12822.558181272 _step‚1163‹ z /æ4-/ _wandbruntime‚12822  _runtime‚12822F¾y¥ (-ƒP H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2153423468151216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16438261182976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1164 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16437919758848 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16413123516344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 $ _timestamp‚1.784519648983456e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚486949729831424 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2182540094253056 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16437919758848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254589388 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16530406430720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2166101816030720 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2166101833070080 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2182539735789568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16438261182976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58453817 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58454096 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2182540094253056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33768044 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚486949605949952 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254579243 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ñyÄÉ9‘-ˆP F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24685773 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2166101833070080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140016253 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚470419323400704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2182539735789568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24685776 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140006110 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33768320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140016253 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114573133 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140006110 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16530301161472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2166101816030720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114573135 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114573135 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚14149 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _runtime‚ 12822.5620024 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2153423451111856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114573133 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58454096 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2182540094253056 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33768044 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚486949605949952 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254579243 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2153423468151216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16438261182976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1164 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16437919758848 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1pŪ²Õ  A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16413123516344 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 $ _timestamp‚1.784519648983456e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚486949729831424 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2182540094253056 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16437919758848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254589388 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16530406430720 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2166101816030720 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2166101833070080 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2182539735789568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16438261182976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58453817 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2166250090532864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140015766 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58457775 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254596692 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2166250090532864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16439371289088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2166250107572224 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114580926 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚486983901829120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16439029857792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚486984025703424 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33770553 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16414231904596 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16439029857792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114580928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845196495855763e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254606851 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140025923 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚14149 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _step‚1165 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58458063 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2169985471284670 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140015766 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚470452503670784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ìþDÇ'”-•N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254606851 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254596692 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16531416770560 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚14149 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _step‚1165 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58458063 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2169985471284670 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140015766 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚470452503670784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16439371289088 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16414572465982 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2182689478861312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2153570881779328 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114580926 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2166250090532864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140015766 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58457775 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254596692 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2166250090532864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16439371289088 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2166250107572224 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114580926 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚486983901829120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16439029857792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚486984025703424 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33770553 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16414231904596 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16439029857792 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114580928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845196495855763e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254606851 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140025923 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EdÈ›-‚(:’ ò·öÒýâ‘Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚124.34' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.44666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›¢únœ-‚(:ô ¸öÒÏŽ¬Ë# disk.dm-0.out‚109602.8466796875 disk.dm-1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084( disk.nvme0c0n1.out‚85170.21142578125' disk.nvme1c1n1.out‚40773.7099609375 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚40773.7099609375 disk.dm-0.in‚ 539.2421875 cpu‚25.68545112214927" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 640.90625 proc.cpu.threads‚17 network.recv‚ 3.346879e+06) disk./.usagePercent‚76.22117472867433& disk.nvme0n1.out‚85170.21142578125# disk.dm-1.out‚15005.97021484375 disk.rbd0.out‚0$ memory_percent‚97.01447486877441' proc.memory.availableMB‚ 489.1484375 network.sent‚ 2.2784689e+07% disk./.usageGB‚1170.1855850219727 disk.nvme0n1.in‚ 309.37109375ÓS u“-‚(:ˆ ¸öÒ—¡ŽÌ gpu.0.gpu‚55.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚276.081' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.027 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åv4ž-/ _wandbruntime‚12864  _runtime‚12864Ù¤ßËŸ-Å _runtime‚12864.831028594 _step‚1167 ' lr-AdamW/pg1‚0.00012351788332348658 ' lr-AdamW/pg2‚0.00012351788332348658 trainer/global_step‚14199 $ _timestamp‚1.784519691253979e+09Þ¯TyР-Ê ' lr-AdamW/pg2‚0.00012351788332348658 trainer/global_step‚14199 $ _timestamp‚1.784519691253979e+09 _runtime‚12864.831028594 _step‚1167 ' lr-AdamW/pg1‚0.00012351788332348658 ^Ü4¡-/ _wandbruntime‚12864  _runtime‚12864]¢?ª® ¢-†P TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16493488860672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140500905 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2173515541137920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58655141 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚14199 _step‚1168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140490762 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚472078336904704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2190009388462080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33882399 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2173515558177280 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2190009029998592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16586168118784 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2177263948815842 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2190009388462080 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2190009029998592 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16468606729752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114962785 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16493488860672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140490762 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _runtime‚12864.831090224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 255453545 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2173515541137920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33882665 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚488664608195584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2160795001524816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2173515558177280 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2177263591215208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 % _timestamp‚1.7845196912546163e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16493830284800 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16468947291026 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255463690 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16586271290880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114962783 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24772473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2190009029998592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16586168118784 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2177263948815842 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2190009388462080 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2190009029998592 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16468606729752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114962785 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16493488860672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140490762 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _runtime‚12864.831090224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 255453545 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2173515541137920 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33882665 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚488664608195584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2160795001524816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2173515558177280 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2177263591215208 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 % _timestamp‚1.7845196912546163e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16493830284800 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16468947291026 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255463690 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16586271290880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114962783 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24772473 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58655141 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚14199 _step‚1168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140490762 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚472078336904704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2190009388462080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33882399 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2173515558177280 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16494598959616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚472111517174784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114970576 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140500418 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58658803 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2190158773070336 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140500418 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16494598959616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24774207 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16469715118004 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16494940390912 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2190158773070336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255470994 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24774210 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33884596 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16470055679390 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2160942415152928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845196918663337e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2173663815640064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114970578 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16587283041792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚14199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255481153 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚472111498562560 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚12865.442065539 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2173663832679424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180ïǼ$¦-•N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚472111498562560 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚12865.442065539 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2173663832679424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2160942432192288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114970576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16494598959616 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚472111517174784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114970576 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140500418 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58658803 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2190158773070336 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140500418 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16494598959616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24774207 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16469715118004 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16494940390912 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2190158773070336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255470994 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24774210 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33884596 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16470055679390 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2160942415152928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845196918663337e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2173663815640064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114970578 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16587283041792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚14199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255481153‘ ®ÎËå4§-/ _wandbruntime‚12865  _runtime‚12865ïR® ¨-‡ 0loss/FBt2/detection_loss‚0.41624367727223327 epoch‚35 trainer/global_step‚14199 0loss/FRt2/assignment_loss‚0.7230481730132806 /loss/FRt1/detection_loss‚0.4616982407188134 $ _timestamp‚1.784519691868408e+09 2loss/SRqqt1/assignment_loss‚0.4547290320613117 2loss/FBt1/assignment_loss‚0.003549355346702786 1loss/SRqqt1/detection_loss‚0.5586838178369719 /loss/FBt1/detection_loss‚0.4973968197898373 0loss/FRt1/assignment_loss‚1.4221999808044719 2loss/SRqqt2/assignment_loss‚0.5065075568339361 %loss/total_loss‚7.144466631673982 _runtime‚12865.444401137 _step‚1170 2loss/FBt2/assignment_loss‚0.004605605549938866 /loss/FRt2/detection_loss‚1.4330531858721316 1loss/SRqqt2/detection_loss‚0.6627511865743522 Juÿ©-Œ 0loss/FRt2/assignment_loss‚0.7230481730132806 /loss/FRt1/detection_loss‚0.4616982407188134 $ _timestamp‚1.784519691868408e+09 2loss/SRqqt1/assignment_loss‚0.4547290320613117 2loss/FBt1/assignment_loss‚0.003549355346702786 1loss/SRqqt1/detection_loss‚0.5586838178369719 /loss/FBt1/detection_loss‚0.4973968197898373 0loss/FRt1/assignment_loss‚1.4221999808044719 2loss/SRqqt2/assignment_loss‚0.5065075568339361 %loss/total_loss‚7.144466631673982 _runtime‚12865.444401137 _step‚1170 2loss/FBt2/assignment_loss‚0.004605605549938866 /loss/FRt2/detection_loss‚1.4330531858721316 1loss/SRqqt2/detection_loss‚0.6627511865743522 0loss/FBt2/detection_loss‚0.41624367727223327 epoch‚35 trainer/global_step‚14199’ rty.ª-‚(:ö ¸öÒ¹øîÍ disk.md0.in‚0$ memory_percent‚97.01340198516846 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15006.08349609375 disk.rbd0.out‚0' disk.nvme1c1n1.out‚40775.3818359375# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17$ disk.nvme0n1.out‚85175.623046875 network.sent‚ 2.2789528e+07 network.recv‚ 3.350542e+06) disk./.usagePercent‚76.22117845578451% disk./.usageGB‚1170.1856422424316& disk.nvme0c0n1.out‚85175.623046875" disk.nvme1c1n1.in‚ 229.87109375$ disk.dm-0.out‚109609.52001953125 disk.nvme0n1.in‚ 309.37109375 disk.md0.out‚ 641.03515625( proc.memory.availableMB‚ 489.32421875) proc.memory.percent‚18.46158504486084 cpu‚25.685185988351307 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚40775.3818359375ÀÈ|L«-‚(:’ ¸öÒó¶‚Í gpu.0.gpu‚20.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚121.657' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.55233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÁC"„¬-‚(:ù Ÿ¸öÒ«ÊË# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17 network.recv‚ 3.354708e+06$ disk.dm-0.out‚109610.17626953125 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375( proc.memory.availableMB‚ 489.13671875 disk.md0.in‚0 disk.dm-1.in‚0& disk.nvme0n1.out‚85176.88232421875( disk.nvme1c1n1.out‚40776.25830078125& disk.nvme1n1.out‚40776.25830078125$ memory_percent‚97.01454639434814) proc.memory.percent‚18.46158504486084 cpu‚25.684941388978313 network.sent‚ 2.2850835e+07) disk./.usagePercent‚76.22118243136869$ disk./.usageGB‚1170.185703277588 disk.md0.out‚ 641.703125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚85176.88232421875 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15006.15380859375" disk.nvme0c0n1.in‚ 309.37109375ùFÜôž­-‚(:“ Ÿ¸öÒ»ñ¦Ì gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚263.272' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.75733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yФÏ}®-‚(:ò ®¸öÒÓ…ÃË" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.01461791992188# proc.memory.availableMB‚489.125 network.recv‚ 3.358604e+06) disk./.usagePercent‚76.22118118899863 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚40780.2255859375 disk.md0.in‚0" disk.dm-0.out‚109611.708984375 disk.rbd0.in‚0# proc.memory.rssMB‚ 3024.74609375 network.sent‚ 2.2857706e+07 disk.dm-1.in‚0 disk.rbd0.out‚0 cpu‚25.684726840774175 proc.cpu.threads‚17% disk./.usageGB‚1170.1856842041016 disk.md0.out‚ 642.1796875# disk.dm-1.out‚15009.43115234375( disk.nvme0c0n1.out‚85178.72998046875& disk.nvme0n1.out‚85178.72998046875' disk.nvme1c1n1.out‚40780.2255859375) proc.memory.percent‚18.46158504486084 disk.dm-0.in‚ 539.2421875ŠNÕ)¯-‚(:’ ®¸öÒ§ˆÌ gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚286.81' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.60333333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛÄ_4°-/ _wandbruntime‚12907  _runtime‚12907 ½ãŸË±-Å _step‚1171 ' lr-AdamW/pg1‚0.00012155021489938042 ' lr-AdamW/pg2‚0.00012155021489938042 trainer/global_step‚14249 $ _timestamp‚1.784519734037791e+09 _runtime‚12907.613919813I¢fÈв-Ê _step‚1171 ' lr-AdamW/pg1‚0.00012155021489938042 ' lr-AdamW/pg2‚0.00012155021489938042 trainer/global_step‚14249 $ _timestamp‚1.784519734037791e+09 _runtime‚12907.613919813“ /öÚ4³-/ _wandbruntime‚12907  _runtime‚12907Z}Ë (´-†P OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2180929283284480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24859176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2197478324207616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33996932 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115352435 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140975414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 136464384 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115352433 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2197478682671104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24859173 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16549057962496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2184690965402850 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2184690607802216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140985557 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚490379316780032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16524430504434 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚12907.614217393 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _step‚1172 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2197478682671104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58856412 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473737331796480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16641984983552 trainer/global_step‚14249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚490379453244416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473737350408704 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2197478324207616 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16549399386624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚307 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115352435 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140985557 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256337992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16524089943160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 117852160 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2168166517859056œE(µ-‹P ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 117852160 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2168166517859056 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2180929283284480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24859176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2197478324207616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33996932 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115352435 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140975414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 136464384 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115352433 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2197478682671104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24859173 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16549057962496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2184690965402850 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2184690607802216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140985557 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚490379316780032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16524430504434 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚12907.614217393 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _step‚1172 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2197478682671104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58856412 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473737331796480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16641984983552 trainer/global_step‚14249 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚490379453244416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473737350408704 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2197478324207616 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16549399386624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚307 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115352435 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140985557 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256337992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16524089943160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336” ͳ{R4¶-/ _wandbruntime‚12908  _runtime‚12908;‰*&Š·-N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58860084 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚490413741860352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2197628067279360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16550509492736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2184839504458686 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140985070 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚14249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256355455 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2168313948526528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16643105919488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16525198331412 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚12908.229033662 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24860907 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256345296 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115360228 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140995227 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845197346532862e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2181077540747264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33999460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2181077557786624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 256345296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16525538892798 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2168313965565888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473770530678784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _step‚1173 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2197628067279360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115360226 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58860370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚490413617986048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256355455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33999177 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24860910 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16550168061440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16550509492736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140985070 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2184839146857940 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2197627708808704 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2181077540747264u;h@'¸-•N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2181077557786624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 256345296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16525538892798 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2168313965565888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473770530678784 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _step‚1173 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2197628067279360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115360226 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58860370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚490413617986048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256355455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33999177 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24860910 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16550168061440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16550509492736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140985070 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2184839146857940 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2197627708808704 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2181077540747264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58860084 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚490413741860352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2197628067279360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16550509492736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2184839504458686 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140985070 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚14249 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256355455 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2168313948526528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16643105919488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16525198331412 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚12908.229033662 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24860907 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256345296 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115360228 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140995227 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845197346532862e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2181077540747264 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33999460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452• *à®4¹-/ _wandbruntime‚12908  _runtime‚12908‚t2¥ º-… 0loss/FRt2/assignment_loss‚0.6486101664638068 0loss/FBt1/assignment_loss‚0.0370177920358047 1loss/SRqqt1/detection_loss‚0.6745410331795495 1loss/SRqqt2/detection_loss‚0.5954276782712313 $loss/total_loss‚7.34915468014535 trainer/global_step‚14249 3loss/SRqqt2/assignment_loss‚0.49729819869457514 0loss/FBt2/detection_loss‚0.48457662200436313 2loss/SRqqt1/assignment_loss‚0.4521545679634823 2loss/FBt2/assignment_loss‚0.007434326191089361 .loss/FRt2/detection_loss‚1.458313516577291 /loss/FBt1/detection_loss‚0.5084509978040141 epoch‚35 /loss/FRt1/detection_loss‚0.5100181923521914 % _timestamp‚1.7845197346543071e+09 _runtime‚12908.229568092 _step‚1174 0loss/FRt1/assignment_loss‚1.4753115886079524±ã¿»-Š 2loss/SRqqt1/assignment_loss‚0.4521545679634823 2loss/FBt2/assignment_loss‚0.007434326191089361 .loss/FRt2/detection_loss‚1.458313516577291 /loss/FBt1/detection_loss‚0.5084509978040141 epoch‚35 /loss/FRt1/detection_loss‚0.5100181923521914 % _timestamp‚1.7845197346543071e+09 _runtime‚12908.229568092 _step‚1174 0loss/FRt1/assignment_loss‚1.4753115886079524 0loss/FRt2/assignment_loss‚0.6486101664638068 0loss/FBt1/assignment_loss‚0.0370177920358047 1loss/SRqqt1/detection_loss‚0.6745410331795495 1loss/SRqqt2/detection_loss‚0.5954276782712313 $loss/total_loss‚7.34915468014535 trainer/global_step‚14249 3loss/SRqqt2/assignment_loss‚0.49729819869457514 0loss/FBt2/detection_loss‚0.48457662200436313– ¿–ë~¼-‚(:ó ½¸öÒåÂ÷Ë" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚40780.9267578125 disk.md0.out‚ 642.57421875" disk.dm-0.out‚109612.638671875) disk./.usagePercent‚76.22120678182185 disk.rbd0.out‚0( disk.nvme0c0n1.out‚85180.01708984375' disk.nvme1c1n1.out‚40780.9267578125 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0) proc.memory.percent‚18.46158504486084 network.sent‚ 2.2873513e+07" disk.nvme0c0n1.in‚ 309.37109375 disk.md0.in‚0# disk.dm-1.out‚15009.61865234375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375$ memory_percent‚97.01480865478516% proc.memory.availableMB‚ 489.09375# proc.memory.rssMB‚ 3024.74609375 cpu‚25.684452851578296 proc.cpu.threads‚17 network.recv‚ 3.361873e+06# disk./.usageGB‚1170.18607711792& disk.nvme0n1.out‚85180.01708984375舽-‚(:’ ½¸öÒÅÚ½Ì gpu.0.gpu‚97.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚122.66' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.88666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„xYƒ¾-‚(:ô ̸öÒäÕ¹Ë proc.cpu.threads‚17 network.sent‚ 2.2923059e+07" disk.nvme0c0n1.in‚ 309.37109375% disk.nvme1n1.out‚40781.4365234375# disk.dm-1.out‚15009.64599609375$ memory_percent‚97.01480865478516% proc.memory.availableMB‚ 489.09375% disk./.usageGB‚1170.1850929260254& disk.nvme0n1.out‚85182.49853515625" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 642.7578125 disk.rbd0.in‚0( disk./.usagePercent‚76.2211426755268( disk.nvme0c0n1.out‚85182.49853515625 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875# disk.dm-0.out‚109615.1923828125 disk.dm-1.in‚0 disk.rbd0.out‚0 network.recv‚ 3.366276e+06' disk.nvme1c1n1.out‚40781.4365234375 disk.md0.in‚0# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 cpu‚25.684221846505785+ÂTáž¿-‚(:“ ̸öÒ¸Û“Ì gpu.0.gpu‚54.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚269.696' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.89866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–^v,À-‚(:ô Û¸öÒ±ÞŸË& disk.nvme0n1.out‚85183.35107421875 proc.cpu.threads‚17( disk.nvme0c0n1.out‚85183.35107421875" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚40782.09619140625 disk.nvme1n1.in‚ 229.87109375( proc.memory.availableMB‚ 489.09765625 network.sent‚ 2.292901e+07$ disk./.usageGB‚1169.946632385254 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme1n1.out‚40782.09619140625) proc.memory.percent‚18.46158504486084# disk.dm-0.out‚109615.8603515625 disk.dm-1.in‚0 disk.dm-1.out‚15009.90234375 disk.nvme0n1.in‚ 309.37109375$ memory_percent‚97.01478481292725# proc.memory.rssMB‚ 3024.74609375 cpu‚25.68399019252055 network.recv‚ 3.369452e+06) disk./.usagePercent‚76.20561031657735 disk.md0.out‚ 643.0234375 disk.dm-0.in‚ 539.2421875v9žÁ-‚(:“ Û¸öÒõ½ñË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚285.326' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.10866666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#öy¸4Â-/ _wandbruntime‚12950  _runtime‚12950tEÓËÃ-Å % _timestamp‚1.7845197765200167e+09 _runtime‚12950.09704652 _step‚1175 ' lr-AdamW/pg1‚0.00011959447328575617 ' lr-AdamW/pg2‚0.00011959447328575617 trainer/global_step‚14299áÐÄ-Ê ' lr-AdamW/pg1‚0.00011959447328575617 ' lr-AdamW/pg2‚0.00011959447328575617 trainer/global_step‚14299 % _timestamp‚1.7845197765200167e+09 _runtime‚12950.09704652 _step‚1175— &R£4Å-/ _wandbruntime‚12950  _runtime‚12950§ãQ…BÆ-†P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24945876 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2175538068272016 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115742085 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2188342991352320 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16604968488448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16697828869120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16604627064320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16604627064320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16579913717842 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115742085 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141460066 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34111660 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24945873 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 trainer/global_step‚14299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2204947618416640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚475396345300480 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16697932041216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845197765206344e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2192117624389224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2188343008391680 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚12950.097109431 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141460066 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2204947976880128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59057816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16579573156568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚1176 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2188343008391680ò¤gT(Ç-‹P TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2192117981989858 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34111940 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2204947618416640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115742083 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚492094295953920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2175538068272016 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115742085 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2188342991352320 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16604968488448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16697828869120 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16604627064320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16604627064320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16579913717842 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115742085 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141460066 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34111660 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24945873 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 trainer/global_step‚14299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2204947618416640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚475396345300480 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16697932041216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845197765206344e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2192117624389224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2188343008391680 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚12950.097109431 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141460066 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2204947976880128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59057816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16579573156568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚1176 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2188343008391680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24945876 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141479879 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2205097361488384 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16605737163264 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24947607 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚475429544182784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚12950.710060474 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16606078594560 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115749876 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2205097361488384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2205097003017728 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2188491265854464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16581022106206 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257229757 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2188491265854464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141479879 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257229757 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2188491282893824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16699048169472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚14299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115749878 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16580681544820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257219598 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2188491282893824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59061578 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34114261 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2192266163444948 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492128466380800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0°zƒ›¸&Ê-•N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16606078594560 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 115749876 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2205097361488384 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2205097003017728 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2188491265854464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16581022106206 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257229757 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2188491265854464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141479879 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257229757 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2188491282893824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16699048169472 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚14299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115749878 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16580681544820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257219598 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2188491282893824 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59061578 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34114261 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2192266163444948 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492128466380800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141479879 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2205097361488384 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16605737163264 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24947607 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚475429544182784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtimÂ44çce‚12950.710060474 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534™ m†•4Ë-/ _wandbruntime‚12950  _runtime‚12950úNZÌ-Š epoch‚35 3loss/SRqqt2/assignment_loss‚0.48681916829877475 2loss/FBt2/assignment_loss‚0.053178897405554176 1loss/SRqqt1/detection_loss‚0.6083247815993713 1loss/SRqqt2/detection_loss‚0.6851030787073292 0loss/FBt1/detection_loss‚0.35844790805670745 /loss/FBt2/detection_loss‚0.7016860838312741 0loss/FRt2/assignment_loss‚0.7277725555943054 3loss/FBt1/assignment_loss‚0.0011564369925334609 /loss/FRt1/detection_loss‚0.4648725858003443 % _timestamp‚1.7845197771357522e+09 3loss/SRqqt1/assignment_loss‚0.39450170741180274 /loss/FRt2/detection_loss‚1.5575467127644302 _runtime‚12950.711029377 /loss/FRt1/assignment_loss‚1.500426709651947 trainer/global_step‚14299 _step‚1178 %loss/total_loss‚7.539836626114374 Æ–ÔÍ- trainer/global_step‚14299 _step‚1178 %loss/total_loss‚7.539836626114374 epoch‚35 3loss/SRqqt2/assignment_loss‚0.48681916829877475 2loss/FBt2/assignment_loss‚0.053178897405554176 1loss/SRqqt1/detection_loss‚0.6083247815993713 1loss/SRqqt2/detection_loss‚0.6851030787073292 0loss/FBt1/detection_loss‚0.35844790805670745 /loss/FBt2/detection_loss‚0.7016860838312741 0loss/FRt2/assignment_loss‚0.7277725555943054 3loss/FBt1/assignment_loss‚0.0011564369925334609 /loss/FRt1/detection_loss‚0.4648725858003443 % _timestamp‚1.7845197771357522e+09 3loss/SRqqt1/assignment_loss‚0.39450170741180274 /loss/FRt2/detection_loss‚1.5575467127644302 _runtime‚12950.711029377 /loss/FRt1/assignment_loss‚1.500426709651947š ’rzýwÎ-‚(:ì ê¸öÒñ©èË network.sent‚ 2.2946734e+07 network.recv‚ 3.372818e+06 disk.dm-0.in‚ 539.2421875 disk.dm-1.out‚ 15010.0078125$ memory_percent‚97.01519012451172# proc.memory.rssMB‚ 3024.74609375% disk./.usageGB‚1169.9465942382812" disk.nvme0n1.out‚ 85186.9921875" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚40782.58642578125 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚40782.58642578125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375 disk.md0.in‚0 disk.md0.out‚ 643.23828125% proc.memory.availableMB‚ 489.03125) proc.memory.percent‚18.46158504486084) disk./.usagePercent‚76.20560783183724$ disk.dm-0.out‚109619.39794921875 disk.dm-1.in‚0 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 85186.9921875 cpu‚25.68369811683911 proc.cpu.threads‚17ÈàŸÏ-‚(:” ê¸öÒˆî¯Ì gpu.0.gpu‚64.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚121.898' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.632666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(5SÐ-‚(:ô ù¸öÒÛïÌË disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚85188.0595703125) proc.memory.percent‚18.46158504486084) disk./.usagePercent‚76.20564311514691" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme0n1.in‚ 309.37109375( proc.memory.availableMB‚ 489.02734375 cpu‚25.683467633985526 proc.cpu.threads‚17 network.sent‚ 2.2995172e+07$ disk./.usageGB‚1169.947135925293 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚40783.158203125 disk.md0.out‚ 643.6484375 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚109620.09326171875# disk.dm-1.out‚15010.08251953125 network.recv‚ 3.377048e+06& disk.nvme1c1n1.out‚40783.158203125 disk.md0.in‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375% disk.nvme0n1.out‚85188.0595703125$ memory_percent‚97.01521396636963# proc.memory.rssMB‚ 3024.74609375?§¾—Ñ-‚(:’ ù¸öÒ¾žÓÌ gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚278.74' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.91333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïžÒzÒ-‚(:ï ˆ¹öÒÙ­ÖË disk.rbd0.in‚0# disk.nvme0n1.out‚85193.76953125" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚109627.23388671875% disk./.usageGB‚1167.9580039978027' disk.nvme1c1n1.out‚40785.0830078125% disk.nvme1n1.out‚40785.0830078125 disk.dm-1.in‚0% proc.memory.availableMB‚ 489.03125 cpu‚25.68321728714909 proc.cpu.threads‚17 network.sent‚ 2.3001966e+07) disk./.usagePercent‚76.07607907492608 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375 disk.md0.out‚ 643.8515625" disk.nvme0c0n1.in‚ 309.37109375% disk.nvme0c0n1.out‚85193.76953125# disk.dm-1.out‚15010.13330078125$ memory_percent‚97.01519012451172# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 network.recv‚ 3.380734e+06ѳžÃžÓ-‚(:“ ˆ¹öÒ˜íšÌ gpu.0.gpu‚76.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚280.931' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.64366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,N?4Ô-/ _wandbruntime‚12992  _runtime‚12992Ý[¾´ÊÕ-Ä ' lr-AdamW/pg1‚0.00011765078409624354 ' lr-AdamW/pg2‚0.00011765078409624354 trainer/global_step‚14349 # _timestamp‚1.78451981872245e+09 _runtime‚12992.300045895 _step‚1179B‹tªÏÖ-É _step‚1179 ' lr-AdamW/pg1‚0.00011765078409624354 ' lr-AdamW/pg2‚0.00011765078409624354 trainer/global_step‚14349 # _timestamp‚1.78451981872245e+09 _runtime‚12992.300045895› Ø\7º4×-/ _wandbruntime‚12992  _runtime‚12992a@` (Ø-…P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚477055358804480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16635056369976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2212416912625664 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2195756716459520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258086596 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16753624613888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141954861 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2212417271089152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59259383 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116131735 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚477055377416704 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16660537590272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2199544640976232 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116131733 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 258076451 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116131735 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚14349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141944718 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚493809098911232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 B÷ÏB(Ù-ŠP aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2182909601645616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚477055358804480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16635056369976 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2212416912625664 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2195756716459520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258086596 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16753624613888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141954861 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2212417271089152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59259383 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116131735 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚477055377416704 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16660537590272 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2199544640976232 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116131733 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 258076451 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116131735 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚14349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141944718 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚493809098911232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2183057032313088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116139528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258093900 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2195905008001024 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258093900 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16754737627648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141964531 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16636505319614 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59263398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141954374 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16661647696384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258104059 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚493843276702208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258104059 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59263120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116139528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚477088539074560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16661306265088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34229088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚477088557686784 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2212566655697408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141954374 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 F=DeviceStù_êatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚1181 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16754842889728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2195904990961664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2183057032313088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116139528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258093900 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2195905008001024 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258093900 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16754737627648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141964531 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16636505319614 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59263398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141954374 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16661647696384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258104059 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚493843276702208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258104059 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59263120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116139528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚477088539074560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16661306265088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34229088 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚477088557686784 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2212566655697408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141954374 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚1181 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16754842889728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2195904990961664 b>y4Ý-/ _wandbruntime‚12992  _runtime‚12992$Ër1Þ-‰ epoch‚35 0loss/FRt1/assignment_loss‚1.4880948540206267 /loss/FRt2/assignment_loss‚0.568954248422695 2loss/FBt1/assignment_loss‚0.014751071725374394 0loss/FRt1/detection_loss‚0.45904758760269915 0loss/FBt2/detection_loss‚0.32206876434266885 &loss/total_loss‚6.9609082443325505 /loss/FRt2/detection_loss‚1.4708698610013784 1loss/SRqqt1/detection_loss‚0.6288629315008062 $ _timestamp‚1.784519819337741e+09 _runtime‚12992.913023724 2loss/SRqqt2/assignment_loss‚0.4836683398714689 1loss/SRqqt2/detection_loss‚0.6251112854804801 2loss/SRqqt1/assignment_loss‚0.3900435861376295 3loss/FBt2/assignment_loss‚0.0027485226984706066 /loss/FBt1/detection_loss‚0.5066871915282517 trainer/global_step‚14349 _step‚1182¯Špbß-Ž _step‚1182 epoch‚35 0loss/FRt1/assignment_loss‚1.4880948540206267 /loss/FRt2/assignment_loss‚0.568954248422695 2loss/FBt1/assignment_loss‚0.014751071725374394 0loss/FRt1/detection_loss‚0.45904758760269915 0loss/FBt2/detection_loss‚0.32206876434266885 &loss/total_loss‚6.9609082443325505 /loss/FRt2/detection_loss‚1.4708698610013784 1loss/SRqqt1/detection_loss‚0.6288629315008062 $ _timestamp‚1.784519819337741e+09 _runtime‚12992.913023724 2loss/SRqqt2/assignment_loss‚0.4836683398714689 1loss/SRqqt2/detection_loss‚0.6251112854804801 2loss/SRqqt1/assignment_loss‚0.3900435861376295 3loss/FBt2/assignment_loss‚0.0027485226984706066 /loss/FBt1/detection_loss‚0.5066871915282517 trainer/global_step‚14349ž f†:†wà-‚(:ì —¹öÒã½êË disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375 disk.md0.out‚ 644.234375 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚109628.19873046875# disk.dm-1.out‚15010.31884765625# disk.nvme0n1.out‚85195.01171875' disk.nvme1c1n1.out‚40785.8115234375 disk.md0.in‚0& proc.memory.availableMB‚ 489.328125) proc.memory.percent‚18.46158504486084 cpu‚25.68296718976631 proc.cpu.threads‚17# disk./.usageGB‚1167.95849609375" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375% disk.nvme1n1.out‚40785.8115234375$ memory_percent‚97.01337814331055# proc.memory.rssMB‚ 3024.74609375 network.sent‚ 2.3017773e+07 disk.dm-1.in‚0% disk.nvme0c0n1.out‚85195.01171875 disk.nvme1n1.in‚ 229.87109375 network.recv‚ 3.384364e+06( disk./.usagePercent‚76.0761111280736 öõHžá-‚(:“ —¹öÒ×£úÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.652' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.55066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëÕáqâ-‚(:ö ¦¹öÒ½Œ’Ë" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚40883.3154296875# memory_percent‚97.0118522644043) proc.memory.percent‚18.46158504486084 disk.rbd0.in‚0 network.sent‚ 2.3067279e+07 network.recv‚ 3.388595e+06 disk.dm-0.in‚ 539.2421875& proc.memory.availableMB‚ 489.578125# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17% disk./.usageGB‚1168.1967887878418 disk.rbd0.out‚0 cpu‚25.682721159085023# disk.dm-1.out‚15010.43994140625) disk./.usagePercent‚76.09163255416654 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 644.57421875# disk.dm-0.out‚109875.4599609375" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚85345.62451171875% disk.nvme1n1.out‚40883.3154296875 disk.dm-1.in‚0 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚85345.62451171875][µGã-‚(:’ ¦¹öÒŒ°òË gpu.0.gpu‚92.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚288.55' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.18333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç{9Þ4ä-/ _wandbruntime‚13034  _runtime‚13034ൻËå-Å ' lr-AdamW/pg1‚0.00011571927217036682 ' lr-AdamW/pg2‚0.00011571927217036682 trainer/global_step‚14399 $ _timestamp‚1.784519861151114e+09 _runtime‚13034.726706867 _step‚1183½=îÐæ-Ê ' lr-AdamW/pg1‚0.00011571927217036682 ' lr-AdamW/pg2‚0.00011571927217036682 trainer/global_step‚14399 $ _timestamp‚1.784519861151114e+09 _runtime‚13034.726706867 _step‚1183Ÿ i [4ç-/ _wandbruntime‚13034  _runtime‚13034ÒUÒ (è-†P A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2219886565298176 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2219886206834688 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16690880144658 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845198611520772e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258960898 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495523844428288 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16809577389056 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16690539583384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2219886206834688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142439513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59460609 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2206972015163874 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2203170458606080 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2203170441566720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59460313 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16716106692096 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2206971657563240 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116521385 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478714372308480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34341040 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2203170458606080 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142439513 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2190281135019216 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16715765267968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25119273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34341333 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116521383 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16809472119808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116521385 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142429370 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2203170441566720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16715765267968 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142429370 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0¶a [ù é-‹P A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116521385 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142429370 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2203170441566720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16715765267968 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142429370 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2219886565298176 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2219886206834688 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16690880144658 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845198611520772e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258960898 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495523844428288 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16809577389056 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16690539583384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2219886206834688 =7DeviceStats*NÌÝMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142439513 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59460609 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2206972015163874 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2203170458606080 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2203170441566720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59460313 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16716106692096 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2206971657563240 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116521385 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478714372308480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34341040 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2203170458606080 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142439513 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2190281135019216 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16715765267968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25119273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34341333 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116521383 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16809472119808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528  ›¸,ê-‚(:ô µ¹öÒóÂôË$ memory_percent‚97.01204299926758& proc.memory.availableMB‚ 489.546875# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375" disk.nvme0c0n1.in‚ 309.37109375 cpu‚25.68246966022656 proc.cpu.threads‚17 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚85346.68994140625( disk.nvme1c1n1.out‚40883.96240234375 disk.md0.out‚ 644.91796875 disk.dm-1.in‚0 network.sent‚ 2.3086111e+07( disk./.usagePercent‚76.0916402568609$ disk./.usageGB‚1168.196907043457& disk.nvme1n1.out‚40883.96240234375 disk.dm-0.in‚ 539.2421875# disk.dm-0.out‚109876.2607421875! disk.dm-1.out‚15010.595703125 disk.rbd0.in‚0 network.recv‚ 3.391778e+06 disk.md0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚85346.68994140625Ììâgžë-‚(:“ µ¹öÒ¡¶µÌ gpu.0.gpu‚73.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚218.378' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚72.79266666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02K¦Ñ4ì-/ _wandbruntime‚13035  _runtime‚13035MHÒ'í-N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59464312 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258968202 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2203318716068864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _step‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142439026 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2207120196618964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2203318733108224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34343585 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16717216798208 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2220035949906432 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16717216798208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495558264470528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16716875366912 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142449183 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116529176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2203318716068864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478747571190784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258978361 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258968202 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚14399 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142449183 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2203318733108224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478747552578560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚13035.342406227 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚495558142693376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25121007 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258978361 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16810693279744 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16810590114816 % _timestamp‚1.7845198617666492e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25121010 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59464595 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116529178 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116529176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34343305 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142439026 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16691647971636ü¶ãÑ'î-•N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16716875366912 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142449183 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116529176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2203318716068864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚478747571190784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258978361 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258968202 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚14399 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142449183 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2203318733108224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478747552578560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚13035.342406227 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚495558142693376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25121007 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258978361 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16810693279744 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16810590114816 % _timestamp‚1.7845198617666492e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25121010 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59464595 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116529178 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116529176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34343305 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142439026 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16691647971636 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59464312 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258968202 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2203318716068864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _step‚1185 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142439026 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2207120196618964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2203318733108224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34343585 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16717216798208 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2220035949906432 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16717216798208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495558264470528¡ ÆY”T4ï-/ _wandbruntime‚13035  _runtime‚13035i¤vð-Š 0loss/FRt1/assignment_loss‚1.4857070195792925 0loss/FBt2/detection_loss‚0.26800829429356826 _step‚1186 2loss/SRqqt1/assignment_loss‚0.4246170050575112 3loss/SRqqt2/assignment_loss‚0.44549633434508457 3loss/FBt1/assignment_loss‚0.0060329910152686645 .loss/FRt2/detection_loss‚1.514734938973561 %loss/total_loss‚7.147413755798525 epoch‚35 trainer/global_step‚14399 0loss/FRt2/assignment_loss‚0.7447856897087188 3loss/FBt2/assignment_loss‚8.210894801932578e-05 /loss/FRt1/detection_loss‚0.4706029514426041 /loss/FBt1/detection_loss‚0.4671959920870293 % _timestamp‚1.7845198617676885e+09 1loss/SRqqt1/detection_loss‚0.7019932938973694 1loss/SRqqt2/detection_loss‚0.6181571364504981 _runtime‚13035.346024733Šîñ- _step‚1186 2loss/SRqqt1/assignment_loss‚0.4246170050575112 3loss/SRqqt2/assignment_loss‚0.44549633434508457 3loss/FBt1/assignment_loss‚0.0060329910152686645 .loss/FRt2/detection_loss‚1.514734938973561 %loss/total_loss‚7.147413755798525 epoch‚35 trainer/global_step‚14399 0loss/FRt2/assignment_loss‚0.7447856897087188 3loss/FBt2/assignment_loss‚8.210894801932578e-05 /loss/FRt1/detection_loss‚0.4706029514426041 /loss/FBt1/detection_loss‚0.4671959920870293 % _timestamp‚1.7845198617676885e+09 1loss/SRqqt1/detection_loss‚0.7019932938973694 1loss/SRqqt2/detection_loss‚0.6181571364504981 _runtime‚13035.346024733 0loss/FRt1/assignment_loss‚1.4857070195792925 0loss/FBt2/detection_loss‚0.26800829429356826¢ b9‡’~ò-‚(:ó ĹöÒ÷š‹Ì network.sent‚ 2.3135746e+07 disk.dm-0.in‚ 539.2421875! disk.dm-1.out‚15010.650390625) proc.memory.percent‚18.46158504486084 cpu‚25.685167718421727) disk./.usagePercent‚76.09166535273609& disk.nvme1n1.out‚40885.46826171875 disk.md0.out‚ 645.359375( disk.nvme0c0n1.out‚85355.15771484375" disk.nvme1c1n1.in‚ 229.87109375& proc.memory.availableMB‚ 489.546875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0n1.out‚85355.15771484375( disk.nvme1c1n1.out‚40885.46826171875$ memory_percent‚97.01204299926758# proc.memory.rssMB‚ 3024.74609375 network.recv‚ 3.395224e+06% disk./.usageGB‚1168.1972923278809 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0! disk.dm-0.out‚109885.23046875 disk.nvme0n1.in‚ 309.37109375 proc.cpu.threads‚17(`þó-‚(:’ ĹöÒã÷–Ë gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.758' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.919333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ô/4ô-/ _wandbruntime‚13051  _runtime‚13051ø~;cõ-þz >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.839833676815033 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10308903455734253 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2354610115289688 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23702877759933472 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5885416269302368 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5428366661071777 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.880473792552948 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24770252406597137 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24671345949172974 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3041423261165619 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8361678123474121 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6061961054801941 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5760312676429749 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_4‚1 7 Pu]Ãu!7rity/*FRt0SRqqt*FBt/FBt_purity‚0.9972149133682251 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5026002526283264 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10710074752569199 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_2‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21244466304779053 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07231523096561432 particle/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17306187748908997 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7220715284347534 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2626994550228119 *jet/accuracy_1_of_2‚0.7370851635932922 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046445805579423904 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7112364768981934 /particle/accuracy_3_of_3‚0.9994120597839355 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9966942071914673 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1873949021100998 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09630399942398071 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5571549534797668 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32640159130096436 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1762835681438446 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3992985785007477 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24770252406597137 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6717050075531006 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6093226671218872 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2642238140106201 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5724599957466125 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0498688668012619 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 *jet/accuracy_2_of_3‚0.5129721760749817 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10825347900390625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 trainer/global_step‚14399 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.588703989982605 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.631380021572113 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5814464092254639 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3879372179508209 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8338812589645386 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6587539315223694 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24012109637260437 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8684359192848206 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.556535542011261 *jet/accuracy_3_of_4‚0.0975145474076271 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6888286471366882 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 _runtime‚13051.207942751 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968740344047546 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4518120586872101 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4985625743865967 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6321468949317932 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8434240221977234 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 *jet/accuracy_2_of_2‚0.1496424674987793 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2483818382024765 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5214964747428894 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6924916505813599 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6183333396911621 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960261583328247 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23406989872455597 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5814313292503357 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7112364768981934 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10308903455734253 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963340759277344 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2234308123588562 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6549144983291626 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03534568473696709 (particle/accuracy‚0.8492683172225952 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2913157045841217 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6070460081100464 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8401172161102295 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2603253126144409 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2647852599620819 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10791049897670746 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7602986097335815 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1985684633255005 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6981918811798096 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_3‚0.9093312621116638 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6113858222961426 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5795397758483887 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5396443605422974 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.225792795419693 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6679645776748657 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6216868162155151 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5392925143241882 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24911701679229736 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6673593521118164 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3033134341239929 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3206551969051361 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5585920214653015 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_3_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30851030349731445 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6325898170471191 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9895060658454895 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 epoch‚35 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6429017782211304 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +particle/specificity‚0.9086534380912781 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900172352790833 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03743920847773552 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2880679666996002 particle/accuracy_3_of_5‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_1_of_5‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5609732270240784 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2516666650772095 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.6430176496505737 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10693053156137466 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7786458134651184 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.760286808013916 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9177476167678833 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6593142151832581 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6383965611457825 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_2‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.26022276282310486 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7029627561569214 6validation_average_jet_accuracy‚0.3460502624511719 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9423472881317139 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5987969636917114 *jet/accuracy_1_of_4‚0.9647114276885986 /particle/accuracy_4_of_4‚0.9514349699020386 _step‚1187 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5243411064147949 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3094368875026703 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6624798774719238 +jet/accuracy_3_of_3‚0.07409610599279404 )loss/val_total_loss‚7.440624341266282 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 $ _timestamp‚1.784519877631973e+09 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5392925143241882 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6015878915786743 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2940373122692108 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26971402764320374 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7009270787239075 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24633146822452545‚÷I‰=ö-ƒ{ A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2647852599620819 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10791049897670746 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7602986097335815 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1985684633255005 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6981918811798096 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_3‚0.9093312621116638 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6113858222961426 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5795397758483887 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5396443605422974 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.225792795419693 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6679645776748657 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6216868162155151 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5392925143241882 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24911701679229736 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6673593521118164 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.3033134341239929 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3206551969051361 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5585920214653015 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_3_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30851030349731445 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6325898170471191 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9895060658454895 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 epoch‚35 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6429017782211304 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +particle/specificity‚0.9086534380912781 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900172352790833 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03743920847773552 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2880679666996002 particle/accuracy_3_of_5‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_1_of_5‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5609732270240784 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.2516666650772095 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.6430176496505737 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10693053156137466 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7786458134651184 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.760286808013916 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9177476167678833 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6593142151832581 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6383965611457825 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_2‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.26022276282310486 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7029627561569214 6validation_average_jet_accuracy‚0.3460502624511719 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9423472881317139 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5987969636917114 *jet/accuracy_1_of_4‚0.9647114276885986 /particle/accuracy_4_of_4‚0.9514349699020386 _step‚1187 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5243411064147949 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3094368875026703 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6624798774719238 +jet/accuracy_3_of_3‚0.07409610599279404 )loss/val_total_loss‚7.440624341266282 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 $ _timestamp‚1.784519877631973e+09 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5392925143241882 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6015878915786743 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2940373122692108 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26971402764320374 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.7009270787239075 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24633146822452545 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.839833676815033 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10308903455734253 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2354610115289688 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23702877759933472 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5885416269302368 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5428366661071777 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.880473792552948 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.24770252406597137 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24671345949172974 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3041423261165619 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8361678123474121 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.6061961054801941 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5760312676429749 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9972149133682251 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.5026002526283264 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10710074752569199 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_2‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21244466304779053 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07231523096561432 particle/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17306187748908997 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7220715284347534 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2626994550228119 *jet/accuracy_1_of_2‚0.7370851635932922 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.046445805579423904 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7112364768981934 /particle/accuracy_3_of_3‚0.9994120597839355 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9966942071914673 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1873949021100998 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09630399942398071 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5571549534797668 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32640159130096436 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1762835681438446 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3992985785007477 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.24770252406597137 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6717050075531006 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6093226671218872 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2642238140106201 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5724599957466125 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0498688668012619 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 *jet/accuracy_2_of_3‚0.5129721760749817 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10825347900390625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 trainer/global_step‚14399 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.588703989982605 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.631380021572113 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5814464092254639 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3879372179508209 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8338812589645386 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6587539315223694 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24012109637260437 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8684359192848206 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.556535542011261 *jet/accuracy_3_of_4‚0.0975145474076271 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6888286471366882 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 _runtime‚13051.207942751 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968740344047546 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4518120586872101 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4985625743865967 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6321468949317932 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8434240221977234 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 *jet/accuracy_2_of_2‚0.1496424674987793 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2483818382024765 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5214964747428894 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6924916505813599 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6183333396911621 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9960261583328247 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23406989872455597 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5814313292503357 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7112364768981934 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10308903455734253 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9963340759277344 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2234308123588562 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6549144983291626 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03534568473696709 (particle/accuracy‚0.8492683172225952 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2913157045841217 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.6070460081100464 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8401172161102295 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2603253126144409 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923£ ëXžw÷-‚(:ì Ó¹öÒ¨ðÃË) proc.memory.percent‚18.46158504486084$ disk.dm-0.out‚111053.17724609375$ disk.nvme0c0n1.out‚ 86493.9609375 disk.nvme0n1.in‚ 309.37109375 cpu‚25.68524435941279 proc.cpu.threads‚17 disk.rbd0.out‚0 network.sent‚ 2.3187811e+07 network.recv‚ 3.399257e+06$ disk.nvme1n1.out‚40916.431640625 disk.dm-1.in‚0) disk./.usagePercent‚76.17273919261291% disk./.usageGB‚1169.4419784545898 disk.md0.out‚ 646.1328125 disk.dm-0.in‚ 539.2421875" disk.nvme0n1.out‚ 86493.9609375" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.01223373413086# proc.memory.rssMB‚ 3024.74609375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0# disk.dm-1.out‚15010.85107421875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme1c1n1.out‚40916.431640625& proc.memory.availableMB‚ 489.515625swžø-‚(:“ Ó¹öÒºŽÈÍ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.657' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.55233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“Üèrù-‚(:ç â¹öÒÜçË disk.nvme0n1.in‚ 309.37109375$ disk.dm-0.out‚111175.54931640625% disk.nvme0n1.out‚86614.7607421875" disk.nvme1c1n1.in‚ 229.87109375" disk.nvme1n1.out‚ 40918.7890625# memory_percent‚97.0123291015625! proc.memory.availableMB‚489.5 cpu‚25.685011941968373 proc.cpu.threads‚17) disk./.usagePercent‚76.17294095351048% disk./.usageGB‚1169.4450759887695# disk.dm-1.out‚15010.93701171875 disk.rbd0.in‚0$ disk.nvme1c1n1.out‚ 40918.7890625# proc.memory.rssMB‚ 3024.74609375 disk.rbd0.out‚0' disk.nvme0c0n1.out‚86614.7607421875 disk.nvme1n1.in‚ 229.87109375) proc.memory.percent‚18.46158504486084 network.sent‚ 2.3234156e+07 network.recv‚ 3.40359e+06 disk.md0.in‚0 disk.md0.out‚ 646.453125 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375{ižú-‚(:“ â¹öÒþÛ­Ì gpu.0.gpu‚39.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚288.137' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.04566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 =Û­4û-/ _wandbruntime‚13093  _runtime‚13093tµõÅü-Æ ' lr-AdamW/pg1‚0.00011380006156552757 ' lr-AdamW/pg2‚0.00011380006156552757 trainer/global_step‚14449 % _timestamp‚1.7845199197050602e+09 _runtime‚13093.281374168 _step+Kž’‚1188qäH¬Ñý-Ë ' lr-AdamW/pg1‚0.00011380006156552757 ' lr-AdamW/pg2‚0.00011380006156552757 trainer/global_step‚14449 % _timestamp‚1.7845199197050602e+09 _runtime‚13093.281374168 _step‚1188¤ W üù4þ-/ _wandbruntime‚13093  _runtime‚13093"‡{ (ÿ-„P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2212526527588864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16886097777664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2229318162755072 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2212526510549504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117033843 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚480395983087104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _runtime‚13093.281444918 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117033843 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142995516 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2199573924401040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2199573907361680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2212526510549504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16766667206963 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260039502 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59700792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143005659 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260029357 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16791652205568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59701072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34490703 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2212526527588864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚497282177745408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16766326645689 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16791652205568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2216340234007369 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784519919705829e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117033841 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚1189 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2229318162755072 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142995516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16886194658304 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚497282062252544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34490980 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16791993629696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚14449 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2229318521218560 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2229318521218560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488Š©r(€.‰P D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143005659 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2229318521218560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2212526527588864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16886097777664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2229318162755072 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2212526510549504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117033843 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚480395983087104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _runtime‚13093.281444918 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117033843 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142995516 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2199573924401040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2199573907361680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2212526510549504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16766667206963 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260039502 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59700792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143005659 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260029357 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16791652205568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59701072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34490703 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2212526527588864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚497282177745408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16766326645689 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16791652205568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2216340234007369 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784519919705829e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117033841 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚1189 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2229318162755072 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142995516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16886194658304 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚497282062252544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34490980 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16791993629696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚14449 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2229318521218560 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2216488773063093 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16767775595327 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚14449 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117041634 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117041636 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _step‚1190 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2199721355068512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260046806 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59704752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25211823 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260056965 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143015329 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16887312514560 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117041636 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2216488773063093 QýÂrain_batch_end/allocated_bytes.large_pool.freed‚2212674785051648 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚497316475871744 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2199721338029152 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚13093.894216928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34492929 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2229467905826816 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2212674785051648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2212674802091008 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143005172 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2216489130663839 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2212674802091008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260046806 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚497316356191744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2229467905826816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16767775595327 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚14449 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117041634 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117041636 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _step‚1190 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2199721355068512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260046806 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59704752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25211823 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260056965 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143015329 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16887312514560 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117041636 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2236787456964096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2219940235656704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34605318 _runtime‚13135.389243236 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25296789 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117423491 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16821809859097 % _timestamp‚1.7845199618138041e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59902396 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143480168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16847562731520 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117423493 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25296792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚14499 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2223767608195011 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2236787815427584 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34605604 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2206945440735280 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482054977978880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚1193}ª‚0 ’.‹P 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2206945440735280 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482054977978880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚1193 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2223767250594377 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59902107 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260913804 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260903659 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2236787456964096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2219940235656704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34605318 _runtime‚13135.389243236 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25296789 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117423491 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16821809859097 % _timestamp‚1.7845199618138041e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59902396 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143480168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16847562731520 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117423493 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25296792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 trainer/global_step‚14499 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2223767608195011 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2236787815427584 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34605604 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0© Üb‡k4“./ _wandbruntime‚13136  _runtime‚13136†¿Üy'”.N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260931267 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16943167466496 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117431286 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260921108 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16848331406336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59906083 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚14499 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499031344327680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16822918247349 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2220088527198208 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117431284 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482088176861184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2207092871402752 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2220088527198208 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7845199624283721e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59906386 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2207092888442112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2236936841565184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2223915789650101 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143489824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143499981 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2223916147250847 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16848672837632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34607560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25298523 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260921108 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260931267 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16943060107264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16848331406336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2236937200035840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117431286 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143499981 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16848672837632 _step‚1194 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25298526 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚499031218356224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16823258808735 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2236937200035840 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2220088510158848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34607860 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117431284 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59906386 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2207092888442112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2236936841565184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2223915789650101 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143489824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143499981 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2223916147250847 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16848672837632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34607560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25298523 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260921108 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260931267 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16943060107264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16848331406336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2236937200035840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117431286 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143499981 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16848672837632 _step‚1194 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25298526 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚499031218356224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16823258808735 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2236937200035840 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2220088510158848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34607860 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117431284 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260931267 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16943167466496 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117431286 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260921108 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16848331406336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59906083 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚14499 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499031344327680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16822918247349 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ª S ?4–./ _wandbruntime‚13136  _runtime‚13136xÔS—.‰ 2loss/FBt2/assignment_loss‚0.004444092728095939 1loss/SRqqt2/detection_loss‚0.6633028566622958 % _timestamp‚1.7845199624293485e+09 _step‚1195 /loss/FBt2/detection_loss‚0.7048623528398041 0loss/FRt1/assignment_loss‚1.5410150452820277 2loss/SRqqt2/assignment_loss‚0.4644807911947447 /loss/FRt1/detection_loss‚0.4804603858939149 /loss/FRt2/detection_loss‚1.5465798618334183 1loss/SRqqt1/detection_loss‚0.6699656453180824 trainer/global_step‚14499 %loss/total_loss‚7.488054138852515 epoch‚36 0loss/FRt2/assignment_loss‚0.6277898224270714 2loss/SRqqt1/assignment_loss‚0.3510252417063931 4loss/FBt1/assignment_loss‚0.00037019794756996055 /loss/FBt1/detection_loss‚0.4337578450190963 _runtime‚13136.004603193ÔZS†˜.Ž 1loss/SRqqt2/detection_loss‚0.6633028566622958 % _timestamp‚1.7845199624293485e+09 _step‚1195 /loss/FBt2/detection_loss‚0.7048623528398041 0loss/FRt1/assignment_loss‚1.5410150452820277 2loss/SRqqt2/assignment_loss‚0.4644807911947447 /loss/FRt1/detection_loss‚0.4804603858939149 /loss/FRt2/detection_loss‚1.5465798618334183 1loss/SRqqt1/detection_loss‚0.6699656453180824 trainer/global_step‚14499 %loss/total_loss‚7.488054138852515 epoch‚36 0loss/FRt2/assignment_loss‚0.6277898224270714 2loss/SRqqt1/assignment_loss‚0.3510252417063931 4loss/FBt1/assignment_loss‚0.00037019794756996055 /loss/FBt1/detection_loss‚0.4337578450190963 _runtime‚13136.004603193 2loss/FBt2/assignment_loss‚0.004444092728095939« ás-_™.‚(:ô žºöÒ¡íË& disk.nvme1c1n1.out‚41078.541015625 disk.dm-0.in‚ 539.2421875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚86644.49658203125& disk.nvme0n1.out‚86644.49658203125" disk.nvme1c1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3024.74609375 cpu‚25.684072987399915 proc.cpu.threads‚17 network.sent‚ 2.3322129e+07 disk.dm-1.out‚ 15011.3046875" disk.nvme0c0n1.in‚ 309.37109375( proc.memory.availableMB‚ 488.86328125) proc.memory.percent‚18.46158504486084% disk./.usageGB‚1169.5914344787598 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-1.in‚0$ memory_percent‚97.01621532440186 network.recv‚ 3.417703e+06) disk./.usagePercent‚76.18247415591986$ disk.nvme1n1.out‚41078.541015625 disk.md0.out‚ 647.87890625$ disk.dm-0.out‚111361.55712890625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375ç|¦ œš.‚(:‘ žºöÒ™´Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.715' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.23833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0± =ç|›.‚(:ñ ­ºöÒòÑ—Ë$ disk.dm-0.out‚111362.08447265625$ disk.nvme0n1.out‚86645.388671875" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚41079.2080078125 disk.md0.out‚ 648.28125 disk.dm-1.in‚0& disk.nvme0c0n1.out‚86645.388671875 proc.cpu.threads‚17( disk./.usagePercent‚76.1824729135498 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚41079.2080078125 disk.rbd0.out‚0' proc.memory.availableMB‚ 488.5078125# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 network.recv‚ 3.421416e+06% disk./.usageGB‚1169.5914154052734 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 network.sent‚ 2.3371742e+07 disk.nvme0n1.in‚ 309.37109375# disk.dm-1.out‚15011.46630859375$ memory_percent‚97.01838493347168 cpu‚25.683847122528935*:_œ.‚(:’ ­ºöÒîìË gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚268.39' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.46333333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y´P¤.‚(:ô ¼ºöÒûŠÐË% disk.nvme1n1.out‚41080.6416015625( proc.memory.availableMB‚ 488.51171875# proc.memory.rssMB‚ 3024.74609375 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15011.49365234375 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375 cpu‚25.683598255579952 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚ 648.453125$ disk.dm-0.out‚111364.78466796875 disk.nvme0n1.in‚ 309.37109375% disk.nvme0n1.out‚86647.0966796875) proc.memory.percent‚18.46158504486084 network.sent‚ 2.337849e+07( disk./.usagePercent‚76.1826058471461% disk./.usageGB‚1169.5934562683105 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme1c1n1.out‚41080.6416015625 disk.nvme1n1.in‚ 229.87109375$ memory_percent‚97.01836109161377 network.recv‚ 3.424948e+06" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0c0n1.out‚86647.0966796875wœ)žž.‚(:“ ¼ºöÒ÷ÁÌ gpu.0.gpu‚96.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚279.995' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.33166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃÆÌÂ4Ÿ./ _wandbruntime‚13177  _runtime‚13177p}øbÌ .Æ ' lr-AdamW/pg1‚0.00010999903659019718 ' lr-AdamW/pg2‚0.00010999903659019718 trainer/global_step‚14549 % _timestamp‚1.7845200043858964e+09 _runtime‚13177.963026885 _step‚1196¡­ìÑ¡.Ë ' lr-AdamW/pg1‚0.00010999903659019718 ' lr-AdamW/pg2‚0.00010999903659019718 trainer/global_step‚14549 % _timestamp‚1.7845200043858964e+09 _runtime‚13177.963026885 _step‚1196¬ NvµO4¢./ _wandbruntime‚13177  _runtime‚13177švk£.„P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16902790409216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2227353960763904 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16903131833344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60103548 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784520004386507e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16902790409216 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16903131833344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143974963 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2227353960763904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117813143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2244256751173120 P@DeviceSÝ7.’{tatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2244257109636608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2231194267181385 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2227353977803264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚483714010095104 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 261777961 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143964820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2244256751173120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚483713991482880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16997781284352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16997880262144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2244257109636608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34720342 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚14549 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2231194624782019 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2214316974108880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25383489 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117813141 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16877293072505 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2227353977803264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60103834 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34720059 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚500711890357248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _runtime‚13177.963083655 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143974963 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500711772767232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16877633633779 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528á}ãõ(¤.‰P QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2227353977803264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60103834 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34720059 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚500711890357248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 _runtime‚13177.963083655 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143974963 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500711772767232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16877633633779 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16902790409216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2227353960763904 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16903131833344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60103548 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784520004386507e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16902790409216 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16903131833344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143974963 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2227353960763904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117813143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2244256751173120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2244257109636608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2231194267181385 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2227353977803264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚483714010095104 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 261777961 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143964820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2244256751173120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚483713991482880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16997781284352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16997880262144 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2244257109636608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚283 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34720342 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚14549 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2231194624782019 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2214316974108880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25383489 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117813141 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16877293072505 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528­ Ë44ˆ4¥./ _wandbruntime‚13178  _runtime‚13178¹YÒË'¦.N KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34722595 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16904241939456 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117820936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16903900508160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16998997725696 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143984633 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 261795410 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60107536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16998894560768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 261805569 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845200049967308e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16904241939456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2214464421815712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500746188090880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚483747171752960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2214464404776352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143974476 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143974476 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 261805569 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚500746066313728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2244406494244864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2227502252305408 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚13178.572453106 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _step‚1198 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117820934 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2227502235266048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117820934 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2244406494244864 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2227502252305408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25385226 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2227502235266048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25385223 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚483747190365184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16878401460757 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184:µÝ §.•N D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845200049967308e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16904241939456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2214464421815712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500746188090880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚483747171752960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2214464404776352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143974476 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143974476 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 261805569 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚500746066313728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2244406494244864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2227502252305408 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚13178.572453106 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _step‚1198 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117820934 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2227502235266048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117820934 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2244406494244864 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2227502252305408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25385226 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.onÿP _train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2227502235266048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25385223 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚483747190365184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16878401460757 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34722595 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16904241939456 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117820936 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16903900508160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16998997725696 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143984633 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 261795410 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60107536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16998894560768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 261805569 F4´./ _wandbruntime‚13220  _runtime‚13220pžÞ (µ.†P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _runtime‚13220.266152362 _step‚1201 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2234767702910464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118202793 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2221688507482480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2234767702910464 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2251726403845632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16932776285913 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16958359511040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚14599 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2234767685871104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚502426759757824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚485373004986880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2238621641369027 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2221688524521840 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16958700935168 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34834830 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2234767685871104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2251726045382144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25470189 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485373023599104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262662408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25470192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16933116847187 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2251726403845632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144449472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60304731 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60305022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34834542 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17053624598016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118202793 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118202791 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17053736158720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚101453×(¶.‹P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118202791 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17053736158720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _runtime‚13220.266152362 _step‚1201 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2234767702910464 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118202793 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2221688507482480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2234767702910464 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2251726403845632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16932776285913 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16958359511040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚14599 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2234767685871104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚502426759757824 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚485373004986880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2238621641369027 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2221688524521840 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16958700935168 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34834830 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2234767685871104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2251726045382144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25470189 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485373023599104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262662408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25470192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16933116847187 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2251726403845632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144449472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60304731 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60305022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34834542 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17053624598016 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118202793 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288± úbeS4·./ _wandbruntime‚13221  _runtime‚13221ì@ ˜t ¸.N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 $ _timestamp‚1.784520047605177e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262679871 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16959469609984 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2221835955189312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2238770180424863 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16959811041280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262679871 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25471926 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16933884674165 trainer/global_step‚14599 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118210584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34836815 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17054852999680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2251875429983232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚1202 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17054739348992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118210584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262669712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16959469609984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60308738 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25471923 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144459128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16934225235551 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262669712 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22218359381499521òï'¹.”N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144469285 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60309041 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2251875429983232 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚1202 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17054739348992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118210584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262669712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16959469609984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60308738 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25471923 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144459128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16934225235551 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262669712 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2221835938149952 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 $ _timestamp‚1.784520047605177e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262679871 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16959469609984 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2221835955189312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2238770180424863 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16959811041280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262679871 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25471926 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16933884674165 trainer/global_step‚14599 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118210584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34836815 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17054852999680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2259195339591168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚487032037103104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118592441 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚14649 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144934124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25556889 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17014270036992 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2259195339591168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚504141577572864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17014270036992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2229060057895440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118592443 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 263536710 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144944267 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2246048300355401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17013928612864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2229060040856080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _step‚1205 _runtime‚13262.925054275 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118592443 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25556892 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17109540469760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34950085 % _timestamp‚1.7845200893463774e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17109435200512 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2242181410978304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16988259499321 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118592441 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2259195698054656 ]HDeviceStatsMonitor.on_train_batch_start/allocateÖzHŠÕd_bytes.large_pool.freed‚2242181410978304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60506684 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34950085 % _timestamp‚1.7845200893463774e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17109435200512 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2242181410978304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16988259499321 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118592441 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2259195698054656 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2242181410978304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60506684 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2259195339591168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚487032037103104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118592441 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚14649 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144934124 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25556889 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17014270036992 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2259195339591168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚504141577572864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17014270036992 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2229060057895440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118592443 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 263536710 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144944267 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2246048300355401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17013928612864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2229060040856080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _step‚1205 _runtime‚13262.925054275 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118592443 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25556892 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17109540469760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 J4É./ _wandbruntime‚13263  _runtime‚13263NeyK'Ê.N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17015038711808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25558626 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17110657929216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144953937 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118600236 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144953937 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17015380143104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118600234 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144943780 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504175749330944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263544014 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263544014 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2242329685480448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34952346 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144943780 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60510681 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263554173 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60510972 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17110550569984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16989367887573 _runtime‚13263.534364862 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118600236 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2246196839411125 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17015380143104 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2242329685480448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34952058 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7845200899586184e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2246197197011871 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16989708448959 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504175875302400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2242329702519808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118600234 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚487065217373184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2259345082662912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17015038711808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263554173 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2242329702519808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚1206 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2259344724192256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456€;g:'Ë.•N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2259344724192256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17015038711808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25558626 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17110657929216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144953937 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118600236 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144953937 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17015380143104 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118600234 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144943780 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504175749330944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263544014 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263544014 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2242329685480448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34952346 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144943780 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60510681 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263554173 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60510972 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17110550569984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16989367887573 _runtime‚13263.534364862 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118600236 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2246196839411125 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17015380143104 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2242329685480448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34952058 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7845200899586184e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2246197197011871 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16989708448959 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504175875302400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2242329702519808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118600234 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚487065217373184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2259345082662912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17015038711808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263554173 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2242329702519808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚1206¶ gÀÞj4Ì./ _wandbruntime‚13263  _runtime‚13263ú 7Í.ˆ /loss/FRt1/assignment_loss‚1.544761063426645 3loss/FBt2/assignment_loss‚0.0029089010954277297 1loss/SRqqt1/detection_loss‚0.6798208561130832 /loss/FBt2/detection_loss‚0.4266469898383792 trainer/global_step‚14649 % _timestamp‚1.7845200899623916e+09 1loss/SRqqt2/detection_loss‚0.5713622386152708 /loss/FBt1/detection_loss‚0.3910022492716301 3loss/FBt1/assignment_loss‚0.0019298946635686908 %loss/total_loss‚6.980966553207983 epoch‚36 _step‚1207 0loss/FRt2/assignment_loss‚0.6374719876404766 2loss/SRqqt1/assignment_loss‚0.3997924297607648 2loss/SRqqt2/assignment_loss‚0.4268287000878387 /loss/FRt1/detection_loss‚0.4712433318453659 /loss/FRt2/detection_loss‚1.4271979108495323 _runtime‚13263.537779068<ÂzÌ;Î. 3loss/FBt2/assignment_loss‚0.0029089010954277297 1loss/SRqqt1/detection_loss‚0.6798208561130832 /loss/FBt2/detection_loss‚0.4266469898383792 trainer/global_step‚14649 % _timestamp‚1.7845200899623916e+09 1loss/SRqqt2/detection_loss‚0.5713622386152708 /loss/FBt1/detection_loss‚0.3910022492716301 3loss/FBt1/assignment_loss‚0.0019298946635686908 %loss/total_loss‚6.980966553207983 epoch‚36 _step‚1207 0loss/FRt2/assignment_loss‚0.6374719876404766 2loss/SRqqt1/assignment_loss‚0.3997924297607648 2loss/SRqqt2/asµ©Î¾Øsignment_loss‚0.4268287000878387 /loss/FRt1/detection_loss‚0.4712433318453659 /loss/FRt2/detection_loss‚1.4271979108495323 _runtime‚13263.537779068 /loss/FRt1/assignment_loss‚1.544761063426645· TlÓtƒÏ.‚(:ø ¥»öÒ¨Ï¿Ë network.recv‚ 3.450583e+06 disk.nvme0n1.in‚ 309.37109375 disk.rbd0.in‚0$ memory_percent‚97.01647758483887' proc.memory.availableMB‚ 488.8203125 cpu‚25.682223595705686" disk.dm-0.out‚111586.716796875% disk./.usageGB‚1169.7396354675293& disk.nvme0n1.out‚86712.54541015625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 651.00390625" disk.dm-1.out‚15012.2685546875" disk.nvme0c0n1.in‚ 309.37109375 network.sent‚ 2.3539828e+07) disk./.usagePercent‚76.19212737127694( disk.nvme1c1n1.out‚41243.36279296875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚86712.54541015625) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚41243.36279296875 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0# proc.memory.rssMB‚ 3024.74609375i…4 žÐ.‚(:“ ¥»öÒØ“†Ì gpu.0.gpu‚62.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚287.556' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.85199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V²æÑ.‚(:ô ´»öÒꜱË disk.md0.out‚ 651.26953125$ disk.dm-0.out‚111632.27099609375" disk.dm-1.out‚15012.2958984375 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚86735.9892578125 disk.nvme0n1.in‚ 309.37109375$ memory_percent‚97.01652526855469" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚41266.08447265625 disk.dm-1.in‚0# proc.memory.rssMB‚ 3024.74609375 cpu‚25.681999863244446 network.sent‚ 2.3590315e+07 network.recv‚ 3.454605e+06" disk.nvme0c0n1.in‚ 309.37109375 disk.dm-0.in‚ 539.2421875$ proc.memory.availableMB‚488.8125) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17) disk./.usagePercent‚76.00676675239616$ disk./.usageGB‚1166.893886566162% disk.nvme0n1.out‚86735.9892578125( disk.nvme1c1n1.out‚41266.08447265625 disk.md0.in‚0Y«Ô$žÒ.‚(:“ ´»öÒ½¢†Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚268.902' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.63399999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶yÓ.‚(:î ûöÒÞù®Ë) disk./.usagePercent‚76.00672227554806 disk.nvme1n1.in‚ 229.87109375# disk.dm-1.out‚15012.58544921875% disk.nvme0c0n1.out‚86736.91796875 proc.cpu.threads‚17 disk.dm-0.in‚ 539.2421875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375$ proc.memory.availableMB‚488.8125$ disk.dm-0.out‚111632.86083984375 network.sent‚ 2.3593237e+07% disk./.usageGB‚1166.8932037353516 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375$ disk.nvme1n1.out‚41266.845703125 disk.dm-1.in‚0$ memory_percent‚97.01652526855469# disk.nvme0n1.out‚86736.91796875& disk.nvme1c1n1.out‚41266.845703125 disk.md0.in‚0 disk.md0.out‚ 651.64453125 disk.rbd0.in‚0# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 cpu‚25.681832519221853 network.recv‚ 3.457897e+06RÉàHžÔ.‚(:“ ûöÒÒòùË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚275.395' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.79833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äló4Õ./ _wandbruntime‚13305  _runtime‚13305ˆó>ÊÖ.Ä trainer/global_step‚14699 # _timestamp‚1.78452013176174e+09 _runtime‚13305.337343023 _step‚1208 ' lr-AdamW/pg1‚0.00010439281461766823 ' lr-AdamW/pg2‚0.00010439281461766823Á†Ï×.É ' lr-AdamW/pg1‚0.00010439281461766823 ' lr-AdamW/pg2‚0.00010439281461766823 trainer/global_step‚14699 # _timestamp‚1.78452013176174e+09 _runtime‚13305.337343023 _step‚1208¸ DÞw@4Ø./ _wandbruntime‚13305  _runtime‚133059=y‚ (Ù.†P _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17044083274003 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25643592 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17165244545024 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2236431574229680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚488691050607104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145418776 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145418776 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17069839138816 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35064723 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2253475674543043 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35065022 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17165358202880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2236431591269040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2266664633800192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 trainer/global_step‚14699 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2249595136085504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 264400867 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2249595153124864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚488691031994880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118982093 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2249595136085504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2253475316942409 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17043742712729 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145428919 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2249595153124864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118982093 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25643589 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _step‚1209 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 264411012 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2266664992263680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118982091 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145428919 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2266664633800192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118982091 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 264400867 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2249595153124864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚488691031994880 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118982093 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2249595136085504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2253475316942409 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17043742712729 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145428919 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2249595153124864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118982093 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25643589 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _step‚1209 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 264411012 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2266664992263680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118982091 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145428919 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2266664633800192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118982091 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚488691050607104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145418776 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145418776 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17069839138816 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35064723 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2253475674543043 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35065022 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17165358202880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2236431591269040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2266664633800192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180¹ °ÌEÅ4Û./ _wandbruntime‚13305  _runtime‚13305°hà"Ü.N SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25645326 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118989886 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17070949244928 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2266814376871936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 264418316 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2249743410587648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845201323759236e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2249743427627008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚505890699990016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚488724212264960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17166469112832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17070607813632 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚1210 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118989884 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 264418316 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145438589 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17044851100981 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚488724230877184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17070607813632 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118989886 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17166361753600 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118989884 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17045191662367 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2236579021936512 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145438589 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 264428475 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35066910 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.free:ÙVëþ d‚2236579004897152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚505890574018560 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 264428475 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60712233 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2266814376871936 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845201323759236e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2249743427627008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚505890699990016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚488724212264960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17166469112832 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17070607813632 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚1210 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118989884 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 264418316 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145438589 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17044851100981 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚488724230877184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17070607813632 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118989886 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17166361753600 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118989884 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17045191662367 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2236579021936512 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145438589 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 264428475 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35066910 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2236579004897152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚505890574018560 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 264428475 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60712233 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2266814376871936 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0º íýb‘4Þ./ _wandbruntime‚13305  _runtime‚13305í?Í ß.ƒ 2loss/FBt1/assignment_loss‚0.003287085139609634 /loss/FRt2/detection_loss‚1.4932468549346074 /loss/FBt1/detection_loss‚0.4111630230706752 /loss/FBt2/detection_loss‚0.5596985966292058 .loss/FRt1/assignment_loss‚1.61475515005227 0loss/FRt2/assignment_loss‚0.7043833172501032 1loss/SRqqt2/detection_loss‚0.6100369270697358 %loss/total_loss‚7.320868937285894 epoch‚36 $ _timestamp‚1.784520132376954e+09 _runtime‚13305.952254783 /loss/FRt1/detection_loss‚0.4460785299734076 trainer/global_step‚14699 2loss/SRqqt1/assignment_loss‚0.4440177250744449 3loss/SRqqt2/assignment_loss‚0.48255795323889417 1loss/FBt2/assignment_loss‚0.00717651300813894 0loss/SRqqt1/detection_loss‚0.544467261844802 _step‚1211oq³à.ˆ 0loss/FRt2/assignment_loss‚0.7043833172501032 1loss/SRqqt2/detection_loss‚0.6100369270697358 %loss/total_loss‚7.320868937285894 epoch‚36 $ _timestamp‚1.784520132376954e+09 _runtime‚13305.952254783 /loss/FRt1/detection_loss‚0.4460785299734076 trainer/global_step‚14699 2loss/SRqqt1/assignment_loss‚0.4440177250744449 3loss/SRqqt2/assignment_loss‚0.48255795323889417 1loss/FBt2/assignment_loss‚0.00717651300813894 0loss/SRqqt1/detection_loss‚0.544467261844802 _step‚1211 2loss/FBt1/assignment_loss‚0.003287085139609634 /loss/FRt2/detection_loss‚1.4932468549346074 /loss/FBt1/detection_loss‚0.4111630230706752 /loss/FBt2/detection_loss‚0.5596985966292058 .loss/FRt1/assignment_loss‚1.61475515005227» iÑ‚{á.‚(:ð Ò»öÒŽ—«Ë) proc.memory.percent‚18.46158504486084 cpu‚25.68164686286117 network.sent‚ 2.3599054e+07 network.recv‚ 3.460513e+06% disk./.usageGB‚1166.8924751281738 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0# disk.dm-1.out‚15012.69091796875) disk./.usagePercent‚76.00667481701181' disk.nvme0c0n1.out‚86747.7666015625% disk.nvme0n1.out‚86747.7666015625" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 652.078125 disk.dm-1.in‚0& proc.memory.availableMB‚ 488.609375# proc.memory.rssMB‚ 3024.74609375 disk.dm-0.out‚ 111775.921875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚41400.10595703125& disk.nvme1n1.out‚41400.10595703125 disk.dm-0.in‚ 539.2421875$ memory_percent‚97.01776504516602_6ažâ.‚(:“ Ò»öÒ“Å€Ì gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚277.513' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.50433333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öúTwã.‚(:ì á»öÒöÔÑË" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.02248573303223 network.recv‚ 3.464037e+06$ disk./.usageGB‚1166.892406463623 disk.dm-1.in‚0( disk.nvme0c0n1.out‚86748.61181640625& disk.nvme0n1.out‚86748.61181640625 disk.md0.out‚652.3125' proc.memory.availableMB‚ 487.8359375) proc.memory.percent‚18.46158504486084" disk.dm-1.out‚15012.7470703125 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚41400.46044921875 disk.dm-0.in‚ 539.2421875# proc.memory.rssMB‚ 3024.74609375 cpu‚25.68146030043162( disk./.usagePercent‚76.0066703444796 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚41400.46044921875 disk.md0.in‚0 disk.dm-0.out‚ 111776.5625 proc.cpu.threads‚17 network.sent‚ 2.3661715e+07 disk.rbd0.out‚0ämŒé‘ä.‚(:† á»öÒöÑ Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.792' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œX ÿ4å./ _wandbruntime‚13347  _runtime‚13347ÃÅ‚Ìæ.Æ _step‚1212 ' lr-AdamW/pg1‚0.00010254997234807976 ' lr-AdamW/pg2‚0.00010254997234807976 trainer/global_step‚14749 % _timestamp‚1.7845201738167093e+09 _runtime‚13347.392251979½LÑç.Ë trainer/global_step‚14749 % _timestamp‚1.7845201738167093e+09 _runtime‚13347.392251979 _step‚1212 ' lr-AdamW/pg1‚0.00010254997234807976 ' lr-AdamW/pg2‚0.00010254997234807976¼ êlL4è./ _wandbruntime‚13347  _runtime‚13347ý¿\Á (é.…P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145913571 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145903428 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚13347.393931702 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17125066816512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2260902333529417 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2243803107603280 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2274134286472704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2274133928009216 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2257008861192704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60908699 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145903428 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2257008878232064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17125408240640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119371743 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2274133928009216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25730292 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚507571228256768 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17221060973568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17099566487411 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚507571106472448 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2260902691130051 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35178410 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2243803124642640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265285314 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265275169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚490350045498880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25730289 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2257008878232064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17125408240640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119371743 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2274133928009216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25730292 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚507571228256768 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17221060973568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17099566487411 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚507571106472448 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2260902691130051 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsM’è¿onitor.on_train_batch_start/segment.large_pool.current‚1180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35178410 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2243803124642640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265285314 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265275169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚490350045498880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25730289 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2260902333529417 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2243803107603280 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2274134286472704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2274133928009216 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2257008861192704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0½ Wjô4ë./ _wandbruntime‚13348  _runtime‚13348ŠØü…'ì.N MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 trainer/global_step‚14749 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2257157152734208 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2257157135694848 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265302777 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60912677 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25732023 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60912962 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119379534 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35180654 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145923241 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚507605524835328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35180936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2243950538270752 _step‚1214 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265302777 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2257157152734208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚507605400961024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119379536 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119379536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚490383244381184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17222280454144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265292618 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2274283671080960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17100674875775 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145913084 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17126518346752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17126176915456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2274283312610304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚490383225768960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2257157135694848 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2243950555310112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17126518346752 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799ƒ‚ki'í.•N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17126176915456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2243950555310112 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17126518346752 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 trainer/global_step‚14749 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2257157152734208 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2257157135694848 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265302777 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60912677 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25732023 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60912962 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119379534 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35180654 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145923241 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚507605524835328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35180936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2243950538270752 _step‚1214 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265302777 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2257157152734208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚507605400961024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119379536 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119379536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚490383244381184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17222280454144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265292618 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2274283671080960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17100674875775 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145913084 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17126518346752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17126176915456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2274283312610304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚490383225768960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2257157135694848 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17180977342464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61110298 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17276881548288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚509286068626944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2268329350116425 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146388080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492009059002880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2264422603339264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17155049700819 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 266159616 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2251174640976880 % _timestamp‚1.7845202160802612e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _step‚1217 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119761391 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚14799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25816992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35293032 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17276991011840 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17180635918336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146388080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17180977342464 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2264422586299904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17180635918336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492009077615104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2281603222218240 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146398223 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚509285940551168 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336•ßHR(ü.‹P JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17154709139545 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146398223 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2264422603339264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119761393 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119761391 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35293306 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17180977342464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61110298 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17276881548288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚509286068626944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2268329350116425 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146388080 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492009059002880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2264422603339264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17155049700819 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 266159616 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2251174640976880 % _timestamp‚1.7845202160802612e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _step‚1217 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119761391 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚14799 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25816992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35293032 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17276991011840 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17180635918336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146388080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17180977342464 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2264422586299904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17180635918336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492009077615104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2281603222218240 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146398223 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚509285940551168Á ÌfX½4ý./ _wandbruntime‚13390  _runtime‚13390H©*'þ.N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119769186 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2281752965289984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25818723 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2251322071644352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 266177079 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚509320241837056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25818726 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35295568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚509320365711360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146397736 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61114017 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35295294 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 266166920 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2264570877841408 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2264570860802048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 266177079 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146407893 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61114294 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚14799 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 $ _timestamp‚1.784520216693942e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119769184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2268477889172149 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119769184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17156158089183 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17181746017280 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17182087448576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17155817527797 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2281752606819328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2268478246772895 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146397736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2264570860802048 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17278002564096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492042257885184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146407893 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492042239272960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17278107826176{üO¹Î ÿ.”N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 266177079 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚509320241837056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25818726 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35295568 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚509320365711360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146397736 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61114017 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35295294 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 266166920 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2264570877841408 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2264570860802048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 266177079 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146407893 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61114294 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚14799 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 $ _timestamp‚1.784520216693942e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119769184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2268477889172149 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119769184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17156158089183 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17181746017280 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17182087448576 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17155817527797 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2281752606819328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2268478246772895 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146397736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2264570860802048 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17278002564096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492042257885184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146407893 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚492042239272960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17278107826176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119769186 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2281752965289984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25818723 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2251322071644352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \ÈÀã4€// _wandbruntime‚13390  _runtime‚13390¡£a¹/‰ % _timestamp‚1.7845202166949406e+09 /loss/FRt1/assignment_loss‚1.524923916757974 0loss/FRt2/assignment_loss‚0.7501635829695558 1loss/SRqqt2/detection_loss‚0.6343584169272762 0loss/FBt1/detection_loss‚0.39570593932658765 trainer/global_step‚14799 _runtime‚13390.270239525 _step‚1219 2loss/SRqqt1/assignment_loss‚0.4530032309148548 3loss/FBt1/assignment_loss‚7.131227432841918e-05 /loss/FBt2/detection_loss‚0.3391114828611191 epoch‚36 4loss/FBt2/assignment_loss‚4.9856020458945124e-05 /loss/FRt1/detection_loss‚0.4727146321947816 2loss/SRqqt2/assignment_loss‚0.4399724871923149 .loss/FRt2/detection_loss‚1.457983204719932 1loss/SRqqt1/detection_loss‚0.6511172203590831 %loss/total_loss‚7.119175282518267å§n ‚/Ž 3loss/FBt1/assignment_loss‚7.131227432841918e-05 /loss/FBt2/detection_loss‚0.3391114828611191 epoch‚36 4loss/FBt2/assignment_loss‚4.9856020458945124e-05 /loss/FRt1/detection_loss‚0.4727146321947816 2loss/SRqqt2/assignment_loss‚0.4399724871923149 .loss/FRt2/detection_loss‚1.457983204719932 1loss/SRqqt1/detection_loss‚0.6511172203590831 %loss/total_loss‚7.119175282518267 % _timestamp‚1.7845202166949406e+09 /loss/FRt1/assignment_loss‚1.524923916757974 0loss/FRt2/assignment_loss‚0.7501635829695558 1loss/SRqqt2/detection_loss‚0.6343584169272762 0loss/FBt1/detection_loss‚0.39570593932658765 trainer/global_step‚14799 _runtime‚13390.270239525 _step‚1219 2loss/SRqqt1/assignment_loss‚0.4530032309148548à YÓ̪ƒ/‚(:ö ¼öÒü³Ë disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚88943.47705078125 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 653.62109375 disk.dm-1.in‚0# disk.dm-1.out‚15013.23876953125 network.sent‚ 2.3736864e+07 network.recv‚ 3.478131e+06 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚88943.47705078125& disk.nvme1c1n1.out‚41464.130859375$ disk.nvme1n1.out‚41464.130859375( proc.memory.availableMB‚ 487.62109375# proc.memory.rssMB‚ 3024.74609375$ disk./.usageGB‚1169.577320098877" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875# disk.dm-0.out‚114031.7646484375) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17) disk./.usagePercent‚76.18155480207633 disk.md0.in‚0$ memory_percent‚97.02379703521729 cpu‚25.681544211623507ë=î„“„/‚(:ˆ ¼öÒŽëôË gpu.0.gpu‚97.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚121.671' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.557 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P.æ¤4…// _wandbruntime‚13406  _runtime‚13406&C6?=†/¹z trainer/global_step‚14799 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6417742967605591 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5947456359863281 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8331444263458252 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.259123295545578 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975888133049011 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6754069924354553 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6667529940605164 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7465419173240662 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 % _timestamp‚1.7845202325329099e+09 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26730969548225403 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2816203832626343 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9134570360183716 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2928250730037689 6validation_average_jet_accuracy‚0.3549063205718994 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5556499361991882 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9113979339599609 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29489001631736755 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995671510696411 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9924222230911255 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7053624987602234 !particle/accuracy‚ 0.849609375 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5719922184944153 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6621655821800232 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2129722535610199 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04062573239207268 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6235339045524597 +jet/accuracy_2_of_2‚0.15151198208332062 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28137335181236267 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2685171067714691 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 epoch‚36 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5942681431770325 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6263077855110168 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9982993602752686 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9592396020889282 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_2_of_5‚1 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.394775927066803 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6479262113571167 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2790161073207855 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6791683435440063 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21000000834465027 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3262086510658264 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7369791865348816 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 +jet/accuracy_3_of_4‚0.09775324165821075 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.489136278629303 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28426510095596313 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7568501830101013 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5714206695556641 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 *jet/accuracy_2_of_3‚0.5188677310943604 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.050608549267053604 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30392125248908997 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2817137539386749 particle/accuracy_2_of_4‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22970683872699738 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21661537885665894 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 jet/accuracy_1_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6665202379226685 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5395451784133911 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6804211735725403 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6012714505195618 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5753509998321533 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5924561619758606 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6747110486030579 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6907193660736084 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8415343761444092 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5551478266716003 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4064827859401703 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05173421651124954 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9977399110794067 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 *jet/accuracy_1_of_1‚0.3144845962524414 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9083762168884277 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5930474400520325 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6541255116462708 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6521824598312378 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7101682424545288 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 particle/accuracy_3_of_4‚1 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25872310996055603 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24648158252239227 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5877870321273804 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993293285369873 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9933216571807861 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20246879756450653 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 'particle/f_score‚0.6444736123085022 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5710136294364929 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09254581481218338 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09318532049655914 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09181635826826096 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09275385737419128 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 *jet/accuracy_3_of_3‚0.0734967365860939 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9975563883781433 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5862499475479126 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 _runtime‚13406.108938193 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1794518679380417 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33675122261047363 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.867914080619812 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5556499361991882 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5734488368034363 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09182149171829224 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7101682424545288 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09182149171829224 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6323848366737366 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 )loss/val_total_loss‚7.443269845470728 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9461588263511658 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2793213725090027 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26730969548225403 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_1_of_2‚0.7428219318389893 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8841996192932129 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5138108730316162 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8213717937469482 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6695404648780823 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5592778325080872 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5992141366004944 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8263605237007141 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4895542562007904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7291454672813416 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17613497376441956 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6123480200767517 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2665072977542877 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24987372756004333 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6097314953804016 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 /particle/accuracy_3_of_3‚0.9994120597839355 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3315296173095703 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_2_of_4‚0.6853317618370056 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10080272704362869 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07919026911258698 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6347432732582092 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6868606209754944 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8273242115974426 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 /particle/accuracy_4_of_4‚0.9491673707962036 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.321796715259552 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03967658802866936 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 _step‚1220 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4502891004085541 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.257778137922287 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24609149992465973 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21919873356819153 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2062135487794876 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 jet/accuracy_3_of_5‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5052083134651184x¦¾R—‡/¾z 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5930474400520325 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6541255116462708 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6521824598312378 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7101682424545288 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 particle/accuracy_3_of_4‚1 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25872310996055603 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24648158252239227 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5877870321273804 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993293285369873 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9933216571807861 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20246879756450653 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 'particle/f_score‚0.6444736123085022 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5710136294364929 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09254581481218338 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09318532049655914 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09181635826826096 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09275385737419128 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 *jet/accuracy_3_of_3‚0.0734967365860939 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9975563883781433 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5862499475479126 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 _runtime‚13406.108938193 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1794518679380417 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33675122261047363 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.867914080619812 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5556499361991882 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5734488368034363 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09182149171829224 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7101682424545288 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09182149171829224 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6323848366737366 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 )loss/val_total_loss‚7.443269845470728 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9461588263511658 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2793213725090027 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26730969548225403 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_1_of_2‚0.7428219318389893 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8841996192932129 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5138108730316162 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8213717937469482 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6695404648780823 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5592778325080872 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5992141366004944 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8263605237007141 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4895542562007904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7291454672813416 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17613497376441956 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6123480200767517 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2665072977542877 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24987372756004333 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6097314953804016 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 /particle/accuracy_3_of_3‚0.9994120597839355 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3315296173095703 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_2_of_4‚0.6853317618370056 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10080272704362869 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07919026911258698 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6347432732582092 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6868606209754944 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8273242115974426 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 /particle/accuracy_4_of_4‚0.9491673707962036 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.321796715259552 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03967658802866936 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 _step‚1220 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4502891004085541 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.257778137922287 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24609149992465973 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21919873356819153 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2062135487794876 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 jet/accuracy_3_of_5‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5052083134651184 trainer/global_step‚14799 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6417742967605591 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5947456359863281 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8331444263458252 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.259123295545578 7 Purity/0FRt*SRqqt*FBt/FBtùÆ]¨­_purity‚0.9975888133049011 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6754069924354553 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6667529940605164 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7465419173240662 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 % _timestamp‚1.7845202325329099e+09 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26730969548225403 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2816203832626343 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9134570360183716 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2928250730037689 6validation_average_jet_accuracy‚0.3549063205718994 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5556499361991882 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.9113979339599609 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29489001631736755 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995671510696411 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9924222230911255 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7053624987602234 !particle/accuracy‚ 0.849609375 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5719922184944153 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6621655821800232 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2129722535610199 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04062573239207268 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6235339045524597 +jet/accuracy_2_of_2‚0.15151198208332062 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.28137335181236267 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2685171067714691 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 epoch‚36 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5942681431770325 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6263077855110168 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9982993602752686 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9592396020889282 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_2_of_5‚1 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.394775927066803 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6479262113571167 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2790161073207855 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6791683435440063 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21000000834465027 +jet/accuracy_4_of_4‚0.01670061983168125 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3262086510658264 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7369791865348816 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 +jet/accuracy_3_of_4‚0.09775324165821075 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.489136278629303 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28426510095596313 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7568501830101013 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5714206695556641 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41898149251937866 *jet/accuracy_2_of_3‚0.5188677310943604 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.050608549267053604 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30392125248908997 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2817137539386749 particle/accuracy_2_of_4‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22970683872699738 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21661537885665894 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 jet/accuracy_1_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6665202379226685 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5395451784133911 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6804211735725403 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6012714505195618 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5753509998321533 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5924561619758606 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6747110486030579 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6907193660736084 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8415343761444092 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5551478266716003 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4064827859401703 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05173421651124954 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9977399110794067 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 *jet/accuracy_1_of_1‚0.3144845962524414 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9083762168884277Ä Vãö|ˆ/‚(:ñ ¬¼öÒ‘ˆçË disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚88957.42626953125 proc.cpu.threads‚17# disk./.usageGB‚1169.58638381958 disk.dm-1.in‚0$ memory_percent‚97.02396392822266% proc.memory.availableMB‚ 487.59375) proc.memory.percent‚18.46158504486084 cpu‚25.683592804528256 disk.dm-0.in‚ 539.2421875! disk.dm-0.out‚114045.63671875 network.sent‚ 2.3812337e+07) disk./.usagePercent‚76.18214517632829' disk.nvme1c1n1.out‚41465.1181640625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 654.1015625" disk.nvme1c1n1.in‚ 229.87109375 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚88957.42626953125# proc.memory.rssMB‚ 3024.74609375# disk.dm-1.out‚15013.26611328125 network.recv‚ 3.482848e+06% disk.nvme1n1.out‚41465.1181640625;dV>’‰/‚(:‡ ¬¼öÒ»ãâË gpu.0.gpu‚26.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.462' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.154 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ–y}Š/‚(:ò »¼öÒëÕÐË network.sent‚ 2.385823e+07 disk.nvme0n1.in‚ 309.37109375 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 3024.74609375) disk./.usagePercent‚76.17921964331377' disk.nvme1c1n1.out‚41876.7021484375 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚41876.7021484375 disk.md0.out‚ 654.40234375 disk.dm-1.in‚0# disk.dm-1.out‚15013.45361328125# memory_percent‚97.0224380493164% disk./.usageGB‚1169.5414695739746" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚89758.85400390625& disk.nvme0n1.out‚89758.85400390625 disk.md0.in‚0 cpu‚25.683816411561548 proc.cpu.threads‚17 network.recv‚ 3.486787e+06" disk.nvme1c1n1.in‚ 229.87109375! disk.dm-0.out‚115257.78515625% proc.memory.availableMB‚ 487.84375) proc.memory.percent‚18.46158504486084:V9ý“‹/‚(:ˆ »¼öÒׇšÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚273.876' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚91.292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©’ÿŒ/‚(:ô ʼöÒÒ­®Ë) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17 disk.nvme0n1.in‚ 309.37109375' disk.nvme1c1n1.out‚41912.4072265625% disk.nvme1n1.out‚41912.4072265625# disk.dm-1.out‚15013.47314453125 cpu‚25.683965576168635 network.recv‚ 3.490384e+06" disk.nvme0c0n1.in‚ 309.37109375$ disk.dm-0.out‚115350.84521484375 disk.dm-1.in‚0 disk.rbd0.in‚0 network.sent‚ 2.3861152e+07) disk./.usagePercent‚76.17914510111024$ disk./.usageGB‚1169.540325164795& disk.nvme0c0n1.out‚89816.904296875$ disk.nvme0n1.out‚89816.904296875" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875 disk.rbd0.out‚0 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 654.70703125$ memory_percent‚97.02248573303223' proc.memory.availableMB‚ 487.8359375# proc.memory.rssMB‚ 3024.74609375—6-#ž/‚(:“ ʼöÒÄÐûË gpu.0.gpu‚91.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.391' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.79700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â{ñ"4Ž// _wandbruntime‚13449  _runtime‚13449žt=ÇÉ/à & lr-AdamW/pg1‚9.890384680846985e-05 & lr-AdamW/pg2‚9.890384680846985e-05 trainer/global_step‚14849 $ _timestamp‚1.784520276376223e+09 _runtime‚13449.951825279 _step‚1221²äP·Î/È _step‚1221 & lr-AdamW/pg1‚9.890384680846985e-05 & lr-AdamW/pg2‚9.890384680846985e-05 trainer/global_step‚14849 $ _timestamp‚1.784520276376223e+09 _runtime‚13449.951825279Å Þ)lË4‘// _wandbruntime‚13449  _runtime‚13449òA9¤ (’/†P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120273851 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25907805 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120273849 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17256864280064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17230496201850 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17256864280064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2277697926560554 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17256522855936 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2273778655282688 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2273778672322048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267238220 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120273851 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚511044064076288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17353518176256 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2291035536602112 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120273849 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25907808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146964369 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2291035178138624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146954226 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚493690669781504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚493690651169280 _step‚1222 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2291035178138624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35443765 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146964369 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2291035536602112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚511044187957760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61351276 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17353412907008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845202763771904e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 267228075 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35443471 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2273778672322048 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2277698284161188 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452P¾‰²&“/‹P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2273778672322048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267238220 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120273851 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚511044064076288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17353518176256 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2291035536602112 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120273849 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25907808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146964369 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2291035178138624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146954226 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚493690669781504 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚493690651169280 _step‚1222 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2291035178138624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35443765 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146964369 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2291035536602112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚511044187957760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61351276 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17353412907008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845202763771904e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 267228075 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35443471 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2273778672322048 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2277698284161188 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120273851 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25907805 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120273849 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17256864280064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17230496201850 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17256864280064 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2277697926560554 RBDeviceSt'‚_atsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17256522855936 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2273778655282688 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232Æ NŒR4”// _wandbruntime‚13450  _runtime‚13450Òu]'•/N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2291184562739712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _step‚1223 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17257974386176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61355569 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35445747 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25909539 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61355286 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2273926929784832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120281644 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267245524 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120281642 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2260614878065536 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17231604590102 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267255683 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146963882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2291184921210368 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0âp'–/•N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2273926929784832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120281644 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267245524 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120281642 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2260614878065536 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17231604590102 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267255683 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146963882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2291184921210368 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2291184562739712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _step‚1223 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17257974386176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61355569 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35445747 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25909539 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61355286 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0Ç ºm¾×4—// _wandbruntime‚13450  _runtime‚13450ÔÄÉù˜/ % _timestamp‚1.7845202769908566e+09 0loss/FRt2/assignment_loss‚0.7307579286360855 1loss/SRqqt1/detection_loss‚0.7346347041802378 epoch‚37 _runtime‚13450.566192805 _step‚1224 0loss/FRt1/assignment_loss‚1.5191001132479869 /loss/FRt1/detection_loss‚0.4545283303231038 %loss/total_loss‚7.183462406408337 3loss/SRqqt1/assignment_loss‚0.41866529670534325 4loss/FBt1/assignment_loss‚0.00010687754842923134 3loss/SRqqt2/assignment_loss‚0.46972177455903646 3loss/FBt2/assignment_loss‚0.0008373526444827149 /loss/FRt2/detection_loss‚1.4730904196827945 1loss/SRqqt2/detection_loss‚0.5725050630091235 /loss/FBt1/detection_loss‚0.4961798106165776 0loss/FBt2/detection_loss‚0.31333473525513467 trainer/global_step‚14849L+m™™/’ 3loss/SRqqt1/assignment_loss‚0.41866529670534325 4loss/FBt1/assignment_loss‚0.00010687754842923134 3loss/SRqqt2/assignment_loss‚0.46972177455903646 3loss/FBt2/assignment_loss‚0.0008373526444827149 /loss/FRt2/detection_loss‚1.4730904196827945 1loss/SRqqt2/detection_loss‚0.5725050630091235 /loss/FBt1/detection_loss‚0.4961798106165776 0loss/FBt2/detection_loss‚0.31333473525513467 trainer/global_step‚14849 % _timestamp‚1.7845202769908566e+09 0loss/FRt2/assignment_loss‚0.7307579286360855 1loss/SRqqt1/detection_loss‚0.7346347041802378 epoch‚37 _runtime‚13450.566192805 _step‚1224 0loss/FRt1/assignment_loss‚1.5191001132479869 /loss/FRt1/detection_loss‚0.4545283303231038 %loss/total_loss‚7.183462406408337È cC÷¤vš/‚(:ë Ù¼öÒóêþË$ memory_percent‚97.01948165893555& proc.memory.availableMB‚ 488.328125 proc.cpu.threads‚17( disk./.usagePercent‚76.1791339197797 disk.nvme0n1.in‚ 309.37109375% disk.nvme1n1.out‚41913.2158203125 disk.dm-0.in‚ 539.2421875# proc.memory.rssMB‚ 3024.74609375 network.recv‚ 3.493856e+06$ disk./.usageGB‚1169.540153503418 disk.nvme1n1.in‚ 229.87109375" disk.nvme0c0n1.in‚ 309.37109375) proc.memory.percent‚18.46158504486084 cpu‚25.68392524290208$ disk.dm-0.out‚115351.66552734375" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚41913.2158203125 disk.md0.in‚0 disk.md0.out‚ 655.12890625 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 89817.921875 network.sent‚ 2.3879981e+07! disk.nvme0n1.out‚ 89817.921875 disk.dm-1.in‚0# disk.dm-1.out‚15013.56884765625S´‰Ï›/‚(:’ Ù¼öÒмÍÌ gpu.0.gpu‚24.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.824' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.27466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mß H~œ/‚(:ó è¼öÒýØåË disk.nvme0n1.in‚ 309.37109375# disk.nvme1n1.out‚41946.14453125 disk.md0.in‚0$ memory_percent‚97.02105522155762# proc.memory.rssMB‚ 3024.74609375 network.recv‚ 3.498094e+06$ disk.dm-0.out‚115512.79833984375# disk.dm-1.out‚15013.65087890625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚89946.6591796875" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 cpu‚25.683647316488265 network.sent‚ 2.3930291e+07% disk./.usageGB‚1169.7780723571777 disk.rbd0.in‚0% disk.nvme0n1.out‚89946.6591796875% disk.nvme1c1n1.out‚41946.14453125 disk.md0.out‚ 655.32421875' proc.memory.availableMB‚ 488.0703125) disk./.usagePercent‚76.19463099541949 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17 disk.dm-0.in‚ 539.2421875 Uì/‚(:’ è¼öÒâ·ÛË gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚269.51' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.83666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0° 6•ž/‚(:ô ÷¼öÒƒ·€Ì disk.dm-0.in‚ 539.2421875) proc.memory.percent‚18.46158504486084 network.sent‚ 2.3935174e+07) disk./.usagePercent‚76.19463223778955" disk.dm-1.out‚15013.8388671875" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 655.8828125' proc.memory.availableMB‚ 488.3203125# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17 network.recv‚ 3.501231e+06$ disk./.usageGB‚1169.778091430664 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚41974.48388671875& disk.nvme1n1.out‚41974.48388671875 disk.dm-1.in‚0& disk.nvme0c0n1.out‚90025.751953125 disk.md0.in‚0$ disk.dm-0.out‚115618.85693359375$ memory_percent‚97.01952934265137 cpu‚25.683386880709225$ disk.nvme0n1.out‚90025.751953125 disk.nvme1n1.in‚ 229.87109375µ¾éGŸ/‚(:’ ÷¼öÒ·ê¼Ë gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚120.617' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.205666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð%±4 // _wandbruntime‚13492  _runtime‚13492+!DÊ¡/Ä _runtime‚13492.576694971 _step‚1225 & lr-AdamW/pg1‚9.710079772227841e-05 & lr-AdamW/pg2‚9.710079772227841e-05 trainer/global_step‚14899 % _timestamp‚1.7845203190011203e+09€ùÏ¢/É % _timestamp‚1.7845203190011203e+09 _runtime‚13492.576694971 _step‚1225 & lr-AdamW/pg1‚9.710079772227841e-05 & lr-AdamW/pg2‚9.710079772227841e-05 trainer/global_step‚14899É $7&44£// _wandbruntime‚13492  _runtime‚13492se¤/…P YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2298504472347648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17312091957760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120663499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35558876 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2285124943147562 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2281192397429248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61553669 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495349664673280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120663501 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2281192380389888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120663501 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚1226 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17312433381888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845203190020518e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268102377 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2267838980771664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17286319976532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120663499 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147449021 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35559161 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17409328207872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147438878 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 268112522 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17285979415258 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2298504830811136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17312091957760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.\Â%Þ÷on_train_batch_start/allocation.small_pool.allocated‚ 147449021 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚512758992881152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25994505 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2267838963732304 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2298504472347648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2281192380389888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25994508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚495349683285504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2285125300748196 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚13492.577514471UŽ0(¥/ŠP UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2285125300748196 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚13492.577514471 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2298504472347648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17312091957760 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120663499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35558876 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2285124943147562 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2281192397429248 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61553669 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495349664673280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120663501 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2281192380389888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120663501 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _step‚1226 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17312433381888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845203190020518e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268102377 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2267838980771664 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17286319976532 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120663499 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147449021 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35559161 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17409328207872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147438878 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 268112522 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17285979415258 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2298504830811136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17312091957760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147449021 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚512758992881152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25994505 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2267838963732304 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2298504472347648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2281192380389888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25994508 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚495349683285504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452Ê qµÜˆ4¦// _wandbruntime‚13493  _runtime‚134930è‡'§/N D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17313543488000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2285273482203286 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2298654215419392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35561135 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120671292 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17313202056704 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2281340654892032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17287428364896 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2281340671931392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚512793410883584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17410547328000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147458691 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120671294 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2298654215419392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147448534 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17313202056704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268119826 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17410439968768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120671294 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25996239 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268129985 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚13493.192919178 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61557654 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25996242 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2281340654892032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚14899 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61557374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚512793284912128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2285273839804032 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184&ÄxŒ ¨/•N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35561412 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61557654 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25996242 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2281340654892032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚14899 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61557374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚512793284912128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2285273839804032 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17313543488000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2285273482203286 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2298654215419392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35561135 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120671292 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17313202056704 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2281340654892032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17287428364896 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2281340671931392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚512793410883584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17410547328000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147458691 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120671294 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2298654215419392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147448534 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17313202056704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268119826 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚1¦ Áþ7410439968768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 120671294 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25996239 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268129985 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚13493.192919178 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147923530 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121053151 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26081208 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268976679 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17465145957376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121053149 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17465259615232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147933673 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2275210514145264 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121053151 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2288606122536448 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚14949 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2305973766556672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35674412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2305974125020160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61755617 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17367661059584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚514473824134656 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344²g‚Î(·/‹P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17368002483712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2292552317335204 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17341803189940 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2305974125020160 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17341462628666 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2288606122536448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17368002483712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 268986824 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121053149 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17465259615232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147933673 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2275210514145264 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121053151 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2288606122536448 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚14949 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2305973766556672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35674412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2305974125020160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61755617 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17367661059584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚514473824134656 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2305973766556672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121053149 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35674710 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2275210497105904 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚514473956404736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2288606105497088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147923530 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121053151 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26081208 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268976679 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17465145957376 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269004287 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2292700856391040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35676991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147933186 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2306123151157760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268994128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17369112589824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚514508125355008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17466266907648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121060944 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2275357944812736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2288754379999232 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2306123509628416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _runtime‚13535.521751368 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17466374266880 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17342911578304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚497041877059584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121060944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147943343 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17368771158528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147943343 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2306123509628416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 B2l{ÇDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61759637 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121060942 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26082939 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚14949 % _timestamp‚1.7845203619459336e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2288754397038592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17368771158528 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2292700498790294 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2275357927773376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269004287 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232;™þ'º/•N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121060944 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2275357944812736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2288754379999232 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2306123509628416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _runtime‚13535.521751368 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17466374266880 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17342911578304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚497041877059584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121060944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147943343 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17368771158528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147943343 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2306123509628416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61759637 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121060942 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26082939 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 trainer/global_step‚14949 % _timestamp‚1.7845203619459336e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2288754397038592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17368771158528 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2292700498790294 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2275357927773376 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269004287 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269004287 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2292700856391040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35676991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147933186 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2306123151157760 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268994128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17369112589824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚514508125355008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17466266907648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0Ï !^Æ4»// _wandbruntime‚13535  _runtime‚13535vó‡¼/ˆ 0loss/SRqqt2/detection_loss‚0.647256268833695 _runtime‚13535.52304195 1loss/SRqqt1/detection_loss‚0.7429256231314374 epoch‚37 _step‚1232 3loss/SRqqt2/assignment_loss‚0.45829118456996437 3loss/FBt2/assignment_loss‚0.0022855399177653026 /loss/FRt2/detection_loss‚1.6036550466992103 /loss/FBt1/detection_loss‚0.3985929594137885 trainer/global_step‚14949 % _timestamp‚1.7845203619477103e+09 0loss/FRt1/assignment_loss‚1.4822287139003023 0loss/FRt2/assignment_loss‚0.7264880712215717 1loss/SRqqt1/assignment_loss‚0.360439532282148 3loss/FBt1/assignment_loss‚0.0004984403728828878 /loss/FRt1/detection_loss‚0.4491418769411053 0loss/FBt2/detection_loss‚0.45751257654429583 %loss/total_loss‚7.329315833828167Šý/ /loss/FBt1/detection_loss‚0.3985929594137885 trainer/global_step‚14949 % _timestamp‚1.7845203619477103e+09 0loss/FRt1/assignment_loss‚1.4822287139003023 0loss/FRt2/assignment_loss‚0.7264880712215717 1loss/SRqqt1/assignment_loss‚0.360439532282148 3loss/FBt1/assignment_loss‚0.0004984403728828878 /loss/FRt1/detection_loss‚0.4491418769411053 0loss/FBt2/detection_loss‚0.45751257654429583 %loss/total_loss‚7.329315833828167 0loss/SRqqt2/detection_loss‚0.647256268833695 _runtime‚13535.52304195 1loss/SRqqt1/detection_loss‚0.7429256231314374 epoch‚37 _step‚1232 3loss/SRqqt2/assignment_loss‚0.45829118456996437 3loss/FBt2/assignment_loss‚0.0022855399177653026 /loss/FRt2/detection_loss‚1.6036550466992103Ð ÌÐ._ž¾/‚(:“ ³½öÒÌξÌ gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚255.812' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.27066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}9D_~¿/‚(:ó ³½öÒû¸„Í disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 3024.74609375 cpu‚25.682423963363494 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 309.37109375 disk.md0.in‚0 disk.md0.out‚ 657.4296875 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15014.30712890625 network.sent‚ 2.4010427e+07 disk.nvme0n1.in‚ 309.37109375& disk.nvme1c1n1.out‚42066.771484375" disk.dm-0.out‚115887.076171875) proc.memory.percent‚18.46158504486084 network.recv‚ 3.515291e+06% disk./.usageGB‚1170.0170707702637( disk.nvme0c0n1.out‚90205.47021484375& disk.nvme0n1.out‚90205.47021484375 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚42066.771484375$ memory_percent‚97.02010154724121' proc.memory.availableMB‚ 488.2265625( disk./.usagePercent‚76.2101983892046" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-1.in‚0ê}nw}À/‚(:ò ½öÒœªË network.recv‚ 3.519541e+06 disk.dm-1.in‚0% disk.nvme1c1n1.out‚42069.76953125 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17$ disk./.usageGB‚1170.020606994629 disk.rbd0.out‚0( disk.nvme0c0n1.out‚90209.75341796875 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚90209.75341796875 disk.md0.in‚0$ memory_percent‚97.02048301696777# disk.dm-1.out‚15014.40869140625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375# disk.nvme1n1.out‚42069.76953125' proc.memory.availableMB‚ 488.1640625 cpu‚25.682210737750186 network.sent‚ 2.4074697e+07) disk./.usagePercent‚76.21042872461351" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 658.02734375" disk.dm-0.out‚115892.970703125) proc.memory.percent‚18.46158504486084«Vl5œÁ/‚(:‘ ½öÒ˜ÊóË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚120.508' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.16933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VK[Â/‚(:ô ѽöҪѹË( disk.nvme1c1n1.out‚42070.26123046875' proc.memory.availableMB‚ 488.1640625) proc.memory.percent‚18.46158504486084% disk./.usageGB‚1170.0511894226074 disk.dm-1.in‚0# disk.dm-1.out‚15014.50341796875" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0c0n1.out‚90214.333984375# proc.memory.rssMB‚ 3024.74609375 cpu‚25.681972369653202 proc.cpu.threads‚17 disk.rbd0.in‚0& disk.nvme1n1.out‚42070.26123046875 disk.md0.out‚ 658.3359375 disk.dm-0.in‚ 539.2421875" disk.dm-0.out‚115897.271484375$ memory_percent‚97.02048301696777) disk./.usagePercent‚76.21242074076584 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 network.sent‚ 2.4077661e+07 network.recv‚ 3.522772e+06 disk.rbd0.out‚0$ disk.nvme0n1.out‚90214.333984375" disk.nvme1c1n1.in‚ 229.87109375Üö W“Ã/‚(:ˆ ѽöÒþЋÌ gpu.0.gpu‚52.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚123.123' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚41.041 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CAüa4Ä// _wandbruntime‚13577  _runtime‚13577»+¥yÊÅ/Ä _step‚1233 & lr-AdamW/pg1‚9.353530594205192e-05 & lr-AdamW/pg2‚9.353530594205192e-05 trainer/global_step‚14999 % _timestamp‚1.7845204041253374e+09 _runtime‚13577.701643553+…ç&ÏÆ/É _runtime‚13577.701643553 _step‚1233 & lr-AdamW/pg1‚9.353530594205192e-05 & lr-AdamW/pg2‚9.353530594205192e-05 trainer/global_step‚14999 % _timestamp‚1.7845204041253374e+09Ñ ·SÎä4Ç// _wandbruntime‚13577  _runtime‚135779ñÊo (È/†P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498667710293504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2282582047518864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2313443060765696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7845204041261063e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35789934 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17397286403348 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148408182 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121442801 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2296019830604288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61957547 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269850981 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17423230161408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17520934957568 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17396945842074 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148408182 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 117852160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121442799 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61957842 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17423230161408 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2313443419229184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2299979333922212 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚516188763103232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚516188626638848 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚13577.701732373 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17423571585536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26167905 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35789642 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2299978976321578 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26167908 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121442799 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148418325Øõ¦É/‹P ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2296019830604288 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121442801 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonito³¼Jk%r.on_train_batch_start/inactive_split.large_pool.current‚3 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2296019847643648 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269861126 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148408182 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121442801 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2296019830604288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61957547 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269850981 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17423230161408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17520934957568 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17396945842074 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148408182 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 117852160 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121442799 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61957842 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17423230161408 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2313443419229184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2299979333922212 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚516188763103232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚516188626638848 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚13577.701732373 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17423571585536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26167905 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35789642 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2299978976321578 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26167908 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121442799 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148418325 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498667710293504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2282582047518864 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2313443060765696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7845204041261063e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35789934 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17397286403348 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35791964 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚498700871951360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61961603 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚516223054127616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121450594 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26169642 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498700890563584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17522163564032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2282729461146976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17424681691648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269868430 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17424681691648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61961873 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269878589 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121450592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17424340260352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26169639 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35792231 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2300127515377302 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2282729478186336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2300127872978048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _step‚1235 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148427995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17398054230326 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚14999 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148427995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2313592803837440 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269878589 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148417838 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚516222921864704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35791964 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚498700871951360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61961603 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚516223054127616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121450594 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26169642 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚498700890563584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17522163564032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2282729461146976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17424681691648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 269868430 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17424681691648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61961873 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269878589 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121450592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17424340260352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26169639 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35792231 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2300127515377302 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2282729478186336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2300127872978048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _step‚1235 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148427995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17398054230326 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚14999 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148427995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2313592803837440 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269878589 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148417838 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚516222921864704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0Ó ²£*4Í// _wandbruntime‚13578  _runtime‚13578 8º˜Î/Š 2loss/SRqqt2/assignment_loss‚0.3997751772066633 /loss/FRt1/detection_loss‚0.4762966610132505 0loss/SRqqt1/detection_loss‚0.646342467338725 _step‚1236 4loss/FBt1/assignment_loss‚0.00011435244179035304 %loss/total_loss‚6.996681727482014 $ _timestamp‚1.784520404739364e+09 _runtime‚13578.314618708 .loss/FRt2/detection_loss‚1.476986239325085 0loss/SRqqt2/detection_loss‚0.634710723378284 0loss/FBt2/detection_loss‚0.36335003046340053 0loss/FRt1/assignment_loss‚1.5139306177656482 0loss/FRt2/assignment_loss‚0.6405453942716122 4loss/FBt2/assignment_loss‚2.9503352184757534e-06 0loss/FBt1/detection_loss‚0.40652994277761945 epoch‚37 trainer/global_step‚14999 3loss/SRqqt1/assignment_loss‚0.43809717116471625÷d?Ï/ 4loss/FBt1/assignment_loss‚0.00011435244179035304 %loss/total_loss‚6.996681727482014 $ _timestamp‚1.784520404739364e+09 _runtime‚13578.314618708 .loss/FRt2/detection_loss‚1.476986239325085 0loss/SRqqt2/detection_loss‚0.634710723378284 0loss/FBt2/detection_loss‚0.36335003046340053 0loss/FRt1/assignment_loss‚1.5139306177656482 0loss/FRt2/assignment_loss‚0.6405453942716122 4loss/FBt2/assignment_loss‚2.9503352184757534e-06 0loss/FBt1/detection_loss‚0.40652994277761945 epoch‚37 trainer/global_step‚14999 3loss/SRqqt1/assignment_loss‚0.43809717116471625 2loss/SRqqt2/assignment_loss‚0.3997751772066633 /loss/FRt1/detection_loss‚0.4762966610132505 0loss/SRqqt1/detection_loss‚0.646342467338725 _step‚1236Ô Aò1yÐ/‚(:î à½öÒ´ÅãË% disk./.usageGB‚1170.1938362121582% disk.nvme0n1.out‚90232.9345703125" disk.nvme1n1.out‚ 42070.9296875 disk.dm-1.in‚0 disk.rbd0.out‚0) proc.memory.percent‚18.46158504486084" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0c0n1.out‚90232.9345703125 disk.nvme0n1.in‚ 309.37109375 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15014.58154296875# proc.memory.rssMB‚ 3024.74609375 cpu‚25.68173370941906 disk.md0.out‚ 658.51171875 network.sent‚ 2.409661e+07 network.recv‚ 3.526269e+06" disk.nvme1c1n1.in‚ 229.87109375$ disk.nvme1c1n1.out‚ 42070.9296875 disk.nvme1n1.in‚ 229.87109375# memory_percent‚97.0221996307373' proc.memory.availableMB‚ 487.8828125) disk./.usagePercent‚76.22171217796178$ disk.dm-0.out‚115916.05517578125 disk.rbd0.in‚0 proc.cpu.threads‚17Þ(¤qžÑ/‚(:“ à½öÒúã±Ì gpu.0.gpu‚62.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚264.551' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.18366666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iþyzÒ/‚(:ó ï½öÒ˜·ÇË) disk./.usagePercent‚76.22129374772597$ disk./.usageGB‚1170.187412261963 disk.md0.outg6þ‚ 658.87109375 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15014.68798828125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375 network.sent‚ 2.4144177e+07$ disk.dm-0.out‚116091.84033203125" disk.nvme1c1n1.in‚ 229.87109375( proc.memory.availableMB‚ 488.12890625# proc.memory.rssMB‚ 3024.74609375 cpu‚25.681461037456284 proc.cpu.threads‚17 network.recv‚ 3.529654e+06 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚90234.244140625$ disk.nvme0n1.out‚90234.244140625& disk.nvme1c1n1.out‚42246.310546875$ memory_percent‚97.02069759368896 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚42246.310546875) proc.memory.percent‚18.46158504486084[×jœÓ/‚(:‘ ï½öÒ·à˜Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.922' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.30733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åƒô€Ô/‚(:õ þ½öҞ½Ë# proc.memory.rssMB‚ 3024.74609375 cpu‚25.68129841097701 network.sent‚ 2.4150984e+07 network.recv‚ 3.533053e+06( disk./.usagePercent‚76.2212902690898 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚90236.67822265625$ disk.nvme1n1.out‚42247.126953125# disk.dm-0.out‚116093.9970703125& disk.nvme1c1n1.out‚42247.126953125) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 659.33984375 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15014.78955078125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚90236.67822265625 disk.nvme0n1.in‚ 309.37109375% disk./.usageGB‚1170.1873588562012" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.02088832855225( proc.memory.availableMB‚ 488.09765625·áÔžÕ/‚(:“ þ½öÒÛʇÌ gpu.0.gpu‚91.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚263.182' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.72733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å6ÚŽ4Ö// _wandbruntime‚13620  _runtime‚13620íO¨Ê×/Ä trainer/global_step‚15049 % _timestamp‚1.7845204466752746e+09 _runtime‚13620.250845412 _step‚1237 & lr-AdamW/pg1‚9.177309225289143e-05 & lr-AdamW/pg2‚9.177309225289143e-05¯I”ÏØ/É _step‚1237 & lr-AdamW/pg1‚9.177309225289143e-05 & lr-AdamW/pg2‚9.177309225289143e-05 trainer/global_step‚15049 % _timestamp‚1.7845204466752746e+09 _runtime‚13620.250845412Õ "e³r4Ù// _wandbruntime‚13620  _runtime‚13620€DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17452769616756 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17452429055482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚1238 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2303433572750848 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2320912354974720 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2320912354974720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚15049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2303433572750848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500326705185280 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517903573313024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2307406350509220 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2289953563853104 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17478799263232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35905285 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2303433555711488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121832449 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62159893 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17576742149120 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2320912713438208 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2320912713438208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17479140687360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148902977 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148892834 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148892834 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17452769616756 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17452429055482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚1238 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2303433572750848 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2320912354974720 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2320912354974720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚15049 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2303433572750848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500326705185280 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517903573313024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2307406350509220 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2289953563853104 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17478799263232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35905285 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2303433555711488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121832449 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62159893 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17576742149120 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2320912713438208Ö T”&4Ü// _wandbruntime‚13620  _runtime‚13620º·™œ#Ý/N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121840242 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62163567 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26256339 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35907228 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270742732 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2303581830213632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62163851 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2321062098046464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148902490 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17479909362176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17453537443734 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2303581830213632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148912647 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚15049 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚13620.867125669 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270752891 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2303581847252992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35907509 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500359904067584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17577861352448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2307554889565056 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _step‚1239 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2290101011559936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270742732 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121840244 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 % _timestamp‚1.7845204472913277e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148912647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2321061739575808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26256342 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2321062098046464 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2303581847252992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17480250793472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270752891 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2307554531964310 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62163851 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2321062098046464 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148902490 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17479909362176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17453537443734 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2303581830213632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148912647 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚15049 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚13620.867125669 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270752891 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2303581847252992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35907509 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500359904067584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17577861352448 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2307554889565056 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _step‚1239 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2290101011559936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270742732 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121840244 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 % _timestamp‚1.7845204472913277e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148912647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2321061739575808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26256342 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2321062098046464 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2303581847252992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17480250793472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270752891 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2307554531964310 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62163567 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26256339 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35907228 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270742732 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2303581830213632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528× ‹F¦£4ß// _wandbruntime‚13620  _runtime‚13620fÿ^à/‹ 0loss/FRt1/assignment_loss‚1.4542686134493317 2loss/SRqqt2/assignment_loss‚0.5192835475690419 1loss/SRqqt1/detection_loss‚0.6282189043666042 .loss/FBt1/detection_loss‚0.387685352385742 epoch‚37 2loss/SRqqt1/assignment_loss‚0.4591947897946699 2loss/FBt1/assignment_loss‚0.009902672855281576 trainer/global_step‚15049 0loss/FRt2/assignment_loss‚0.6721635661040893 0loss/FRt1/detection_loss‚0.45214689312732625 /loss/FRt2/detection_loss‚1.4391811799229772 0loss/FBt2/detection_loss‚0.35590040663500194 % _timestamp‚1.7845204472924035e+09 _runtime‚13620.867731989 _step‚1240 4loss/FBt2/assignment_loss‚2.0879696914053255e-05 1loss/SRqqt2/detection_loss‚0.6466737602167518 &loss/total_loss‚7.0246405661237326÷L´á/ &loss/total_loss‚7.0246405661237326 0loss/FRt1/assignment_loss‚1.4542686134493317 2loss/SRqqt2/assignment_loss‚0.5192835475690419 1loss/SRqqt1/detection_loss‚0.6282189043666042 .loss/FBt1/detection_loss‚0.387685352385742 epoch‚37 2loss/SRqqt1/assignment_loss‚0.4591947897946699 2loss/FBt1/assignment_loss‚0.009902672855281576 trainer/global_step‚15049 0loss/FRt2/assignment_loss‚0.6721635661040893 0loss/FRt1/detection_loss‚0.45214689312732625 /loss/FRt2/detection_loss‚1.4391811799229772 0loss/FBt2/detection_loss‚0.35590040663500194 % _timestamp‚1.7845204472924035e+09 _runtime‚13620.867731989 _step‚1240 4loss/FBt2/assignment_loss‚2.0879696914053255e-05 1loss/SRqqt2/detection_loss‚0.6466737602167518Ø é}íuƒâ/‚(:ø ¾öÒñ›ÏË disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375( proc.memory.availableMB‚ 487.81640625# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17 disk.md0.in‚0# disk.dm-0.out‚116095.1298828125 network.sent‚ 2.4166755e+07 network.recv‚ 3.536045e+06) disk./.usagePercent‚76.22128206944741 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚42247.90283203125 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚90237.90380859375$ memory_percent‚97.02260494232178 cpu‚25.681098640264345& disk.nvme1n1.out‚42247.90283203125 disk.md0.out‚ 659.66796875% disk./.usageGB‚1170.1872329711914& disk.nvme0n1.out‚90237.90380859375 disk.dm-1.out‚15014.92578125) proc.memory.percent‚18.46158504486084 disk.dm-0.in‚ 539.2421875¯XÁ‘ã/‚(:† ¾öÒ³‡¥Ì gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.215' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nºnä/‚(:ô œ¾öÒùµË" disk.nvme1c1n1.in‚ 229.87109375 cpu‚25.680918936250098 proc.cpu.threads‚17 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme1n1.out‚42248.69384765625 disk.md0.out‚ 659.90625 disk.dm-0.in‚ 539.2421875) proc.memory.percent‚18.46158504486084% disk./.usageGB‚1170.1877174377441( disk.nvme0c0n1.out‚90240.30810546875# disk.dm-0.out‚116097.7900390625 disk.dm-1.out‚15014.94921875$ memory_percent‚97.02107906341553 network.recv‚ 3.540135e+06 disk.rbd0.in‚0( disk.nvme1c1n1.out‚42248.69384765625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 488.06640625# proc.memory.rssMB‚ 3024.74609375 network.sent‚ 2.4218286e+07( disk./.usagePercent‚76.2213136256469 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚90240.30810546875[ŸEΞå/‚(:“ œ¾öÒ¨¥÷Ë gpu.0.gpu‚91.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚259.744' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.58133333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•±j4æ// _wandbruntime‚13663  _runtime‚13663T­v©Çç/Á % lr-AdamW/pg1‚9.00247179031576e-05 % lr-AdamW/pg2‚9.00247179031576e-05 trainer/global_step‚15099 $ _timestamp‚1.784520489683034e+09 _runtime‚13663.258646074 _step‚1241ÜÓÌè/Æ _runtime‚13663.258646074 _step‚1241 % lr-AdamW/pg1‚9.00247179031576e-05 % lr-AdamW/pg2‚9.00247179031576e-05 trainer/global_step‚15099 $ _timestamp‚1.784520489683034e+09Ù râu–4é// _wandbruntime‚13663  _runtime‚13663”aù (ê/†P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149387629 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17632661669376 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149387629 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17632558497280 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚15099 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122222099 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2310847280818688 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2310847280818688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62362018 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚519618277186560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17534709789184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36020710 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2310847297858048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17534368365056 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62361733 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚501985718689280 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2314833009495594 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2328381649183744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26341305 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚501985737301504 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2328381649183744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2314833367096228 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2297325097226704 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17534368365056 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149377486 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2328382007647232 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17534709789184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122222099 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _runtime‚13663.261868259 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271609730 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚519618398970880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122222101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2297325114266064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7845204896839538e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2328382007647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2310847297858048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _step‚1242 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122222101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360¥¶”që/‹P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149387629 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17632661669376 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149387629 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17632558497280 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 trainer/global_step‚15099 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122222099 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2310847280818688 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2310847280818688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62362018 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚519618277186560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17534709789184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36020710 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2310847297858048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17534368365056 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62361733 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚501985718689280 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2314833009495594 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2328381649183744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26341305 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚501985737301504 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2328381649183744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2314833367096228 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2297325097226704 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17534368365056 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149377486 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2328382007647232 \ODeviceStatsMonitor.on_t‰o‡  rain_batch_start/inactive_split_bytes.large_pool.current‚18612224 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17534709789184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122222099 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _runtime‚13663.261868259 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271609730 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚519618398970880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122222101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2297325114266064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7845204896839538e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2328382007647232 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2310847297858048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _step‚1242 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122222101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180Ú /ÓRÂ4ì// _wandbruntime‚13663  _runtime‚13663=¬nQ'í/N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚502018917571584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17535819895296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2310995555320832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2314981548551318 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271627193 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122229894 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36022752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271617034 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚502018898959360 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149387142 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845204902961404e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26343042 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17633780697600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚15099 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62365791 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2297472527894176 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122229894 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17633671241216 _step‚1243 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271627193 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2328531392255488 _runtime‚13663.871964776 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚519652570200576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2310995555320832 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26343039 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17535819895296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62366086 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17509361218528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344~ F&'î/•N _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2310995572360192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149397299 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚519652570200576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2310995555320832 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26343039 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17535819895296 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62366086 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17509361218528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚502018917571584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17535819895296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2310995555320832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2314981548551318 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271627193 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122229894 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36022752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271617034 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚502018898959360 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149387142 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 % _timestamp‚1.7845204902961404e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26343042 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17633780697600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚15099 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62365791 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2297472527894176 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122229894 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17633671241216 _step‚1243 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271627193 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2328531392255488 _runtime‚13663.871964776 FÀƒñzô/‚(:ï º¾öÒ«ÜÈË$ memory_percent‚97.02517986297607 proc.cpu.threads‚17 network.sent‚ 2.428559e+07 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0 disk.rbd0.out‚0) proc.memory.percent‚18.46158504486084 cpu‚25.680503824654213) disk./.usagePercent‚76.22129300230394( disk.nvme1c1n1.out‚42249.95068359375 disk.dm-0.out‚ 116101.59375! disk.dm-1.out‚15015.134765625" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375 network.recv‚ 3.54789e+06$ disk./.usageGB‚1170.187400817871& disk.nvme1n1.out‚42249.95068359375 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875( disk.nvme0c0n1.out‚90244.47998046875( proc.memory.availableMB‚ 487.39453125# proc.memory.rssMB‚ 3024.74609375" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 660.578125 disk.rbd0.in‚0& disk.nvme0n1.out‚90244.47998046875·• éžõ/‚(:“ º¾öÒ˼’Ì gpu.0.gpu‚95.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚277.794' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.59799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßîÐyö/‚(:î ɾöÒíË network.sent‚ 2.4288554e+07 network.recv‚ 3.55102e+06) proc.memory.percent‚18.46158504486084 cpu‚25.680288978248946# disk.dm-1.out‚15015.23388671875 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚90245.45849609375" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚42250.5048828125 disk.dm-0.in‚ 539.2421875# proc.memory.rssMB‚ 3024.74609375) disk./.usagePercent‚76.22129200840789& disk.nvme0n1.out‚90245.45849609375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 660.953125& proc.memory.availableMB‚ 487.390625$ disk./.usageGB‚1170.187385559082 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375% disk.nvme1n1.out‚42250.5048828125 disk.dm-0.out‚ 116102.21875$ memory_percent‚97.02520370483398 proc.cpu.threads‚17„]О÷/‚(:“ ɾöÒÁêýË gpu.0.gpu‚54.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚288.065' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.02166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uì 4ø// _wandbruntime‚13706  _runtime‚13706%ÒÊù/Ä & lr-AdamW/pg2‚8.829029518766764e-05 trainer/global_step‚15149 % _timestamp‚1.7845205325616212e+09 _runtime‚13706.137251381 _step‚1245 & lr-AdamW/pg1‚8.829029518766764e-050*¦Ïú/É _runtime‚13706.137251381 _step‚1245 & lr-AdamW/pg1‚8.829029518766764e-05 & lr-AdamW/pg2‚8.829029518766764e-05 trainer/global_step‚15149 % _timestamp‚1.7845205325616212e+09Ý ¡ Þ4û// _wandbruntime‚13706  _runtime‚13706³‹EÖ ü/†P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122611751 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149872281 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17688468263936 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26428005 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2318261005925888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚521333088896512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845205325626698e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 272473887 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17589937466880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36135564 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122611751 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.free–QšO6d‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17590278891008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚503644732193280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122611749 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2335850943392768 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2304696630600304 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚521333219069440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚15149 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2318261022965248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2318261022965248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2304696647639664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149872281 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 272484032 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17563736043572 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17589937466880 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2335851301856256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2322260026082602¾ý_(ý/‹P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122611751 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149872281 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17688468263936 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26428005 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2318261005925888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚521333088896512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845205325626698e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 272473887 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17589937466880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36135564 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122611751 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17590278891008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚503644732193280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122611749 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2335850943392768 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2304696630600304 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚521333219069440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚15149 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2318261022965248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2318261022965248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2304696647639664 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149872281 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 272484032 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17563736043572 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17589937466880 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2335851301856256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2322260026082602 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896Þ ü=ùÛ4þ// _wandbruntime‚13706  _runtime‚13706ulx'ÿ/N UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2318409280428032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122619542 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62567505 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 272491336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17591047565824 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122619544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17689471027712 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2336000327993856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚503677912463360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17591047565824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272501495 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚15149 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2322408922739072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17689584678400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2322408565138326 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17564503870550 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2318409280428032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2318409297467392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149881951 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26429742 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36138052 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚521367515753984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272491336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149871794 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2318409297467392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17591388997120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122619542 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149871794 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _step‚1247 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0|.L€0•N VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122619544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36137766 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845205331740158e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17564844431936 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62567505 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 272491336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17591047565824 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122619544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17689471027712 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2336000327993856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚503677912463360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17591047565824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272501495 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚15149 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2322408922739072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17689584678400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2322408565138326 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17564503870550 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2318409280428032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_trainS$¶õÏ_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2318409297467392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149881951 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26429742 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36138052 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚521367515753984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272491336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149871794 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2318409297467392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17591388997120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122619542 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149871794 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _step‚1247 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2318409280428032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122619542 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2343320237601792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150356933 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17645847992832 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150346790 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17744160299008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273358334 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17618878695706 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123001401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17744271859712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123001399 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2325674731033088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62765747 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26514708 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273348189 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505303764309504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36251042 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36251340 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2329687042669610 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17619219256980™:;Ç(0‹P aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚505303745697280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523047905996288 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2329687400270244 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150346790 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17645506568704 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2325674731033088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2325674748072448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26514705 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2343320596065280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2325674748072448 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17645506568704 _runtime‚13748.445962617 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2329687042669610 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17619219256980 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523048036169216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2312068181013264 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123001399 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845205748678472e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2343320237601792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150356933 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17645847992832 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150346790 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17744160299008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273358334 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17618878695706 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123001401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17744271859712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123001399 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2325674731033088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62765747 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26514708 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273348189 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505303764309504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36251042 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36251340 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2329835581725334 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2329835939326080 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273365638 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150356446 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2343469980673536 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123009194 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17745277278208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2312215594641376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150366603 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚505336925967360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2325823005535232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62769724 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123009192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273375797 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36253285 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚523082203245568 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17745388831744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36253575 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚15199 _step‚1251 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2343469622202880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2312215611680736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17620327645344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845205754827774e+09 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17619987083958 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150356446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2325823022574592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _runtime‚13749.058543115 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273375797 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2343469980673536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17646958098944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273365638 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26516439 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17646616667648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17646958098944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62770017 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123009194 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26516442 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452ô`4Å'’0•N QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17646616667648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17646958098944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62770017 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123009194 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26516442 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2329835581725334 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2329835939326080 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273365638 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150356446 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2343469980673536 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123009194 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17745277278208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2312215594641376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150366603 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚505336925967360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2325823005535232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62769724 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123009192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273375797 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36253285 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚523082203245568 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17745388831744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36253575 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚15199 _step‚1251 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2343469622202880 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2312215611680736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17620327645344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845205754827774e+09 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17619987083958 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150356446 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2325823022574592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _runtime‚13749.058543115 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273375797 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2343469980673536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17646958098944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273365638 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26516439ã Ûf44“0/ _wandbruntime‚13749  _runtime‚13749èÏŸ^”0‰ /loss/FBt1/detection_loss‚0.4202467726502667 /loss/FBt2/detection_loss‚0.5597390169020591 _step‚1252 /loss/FRt1/detection_loss‚0.4659608480675031 1loss/SRqqt1/detection_loss‚0.7577153381505488 1loss/SRqqt2/detection_loss‚0.6040599024453973 2loss/SRqqt1/assignment_loss‚0.4147491820929261 2loss/SRqqt2/assignment_loss‚0.4530994936450456 &loss/total_loss‚7.3205459375485855 epoch‚37 % _timestamp‚1.7845205754838195e+09 _runtime‚13749.059229656 0loss/FRt1/assignment_loss‚1.5192216159608127 0loss/FRt2/assignment_loss‚0.6625959309155509 /loss/FRt2/detection_loss‚1.4449954517581551 trainer/global_step‚15199 2loss/FBt1/assignment_loss‚0.014258264859015348 3loss/FBt2/assignment_loss‚0.0039041201013043027éª1}•0Ž 2loss/SRqqt1/assignment_loss‚0.4147491820929261 2loss/SRqqt2/assignment_loss‚0.4530994936450456 &loss/total_loss‚7.3205459375485855 epoch‚37 % _timestamp‚1.7845205754838195e+09 _runtime‚13749.059229656 0loss/FRt1/assignment_loss‚1.5192216159608127 0loss/FRt2/assignment_loss‚0.6625959309155509 /loss/FRt2/detection_loss‚1.4449954517581551 trainer/global_step‚15199 2loss/FBt1/assignment_loss‚0.014258264859015348 3loss/FBt2/assignment_loss‚0.0039041201013043027 /loss/FBt1/detection_loss‚0.4202467726502667 /loss/FBt2/detection_loss‚0.5597390169020591 _step‚1252 /loss/FRt1/detection_loss‚0.4659608480675031 1loss/SRqqt1/detection_loss‚0.7577153381505488 1loss/SRqqt2/detection_loss‚0.6040599024453973ä ±ÑA{–0‚(:ð …¿öÒ‘éÓË( disk.nvme0c0n1.out‚90253.33837890625& disk.nvme0n1.out‚90253.33837890625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0' proc.memory.availableMB‚ 486.9609375% disk./.usageGB‚1170.1875953674316" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375 disk.dm-1.in‚0# disk.dm-1.out‚15018.50830078125# memory_percent‚97.0278263092041" disk.nvme1n1.out‚ 42255.8359375) disk./.usagePercent‚76.22130567447853 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375$ disk.nvme1c1n1.out‚ 42255.8359375# disk.dm-0.out‚116109.3388671875# proc.memory.rssMB‚ 3024.74609375 network.sent‚ 2.4378531e+07 network.recv‚ 3.565412e+06 disk.md0.out‚ 662.2421875 disk.dm-0.in‚ 539.2421875) proc.memory.percent‚18.46158504486084 cpu‚25.680446282457332 proc.cpu.threads‚17{”<œ—0‚(:‘ …¿öÒ™àÌ gpu.0.gpu‚3.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚280.793' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.59766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸN4˜0/ _wandbruntime‚13765  _runtime‚13765ɬL²*™0¸z & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1875000149011612 /particle/accuracy_3_of_3‚0.9994120597839355 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8402493596076965 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34041649103164673 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20431795716285706 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23857714235782623 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27648597955703735 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9923719763755798 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_3_of_4‚1 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚1 )loss/val_total_loss‚7.392778963674386 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34605470299720764 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6765676736831665 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19599315524101257 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8826034069061279 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6598713397979736 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 *jet/accuracy_1_of_4‚0.9750571250915527 6validation_average_jet_accuracy‚0.3654533624649048 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2192557156085968 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.752362072467804 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03853047266602516 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7084606885910034 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6172482371330261 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6568323373794556 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08083568513393402 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _step‚1253 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6844685673713684 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6187575459480286 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07880453765392303 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22092649340629578 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 epoch‚37 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 'particle/f_score‚0.6498445272445679 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5715315937995911 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31550484895706177 +jet/accuracy_3_of_3‚0.07661159336566925 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2005128413438797 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8267385363578796 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4911465048789978 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚13765.434516102 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26180902123451233 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2788272500038147 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6188724040985107 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26285091042518616 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_3‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5139511823654175 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6149253845214844 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7776075005531311 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4613560140132904 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04922683537006378 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08166537433862686 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5786425471305847 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2057950645685196 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5880967378616333 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2095503807067871 *jet/accuracy_1_of_3‚0.9104671478271484 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2501624524593353 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2907833158969879 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9447841644287109 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4873427748680115 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2516079246997833 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6695942878723145 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07806742936372757 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.645946204662323 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.909363865852356 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5882207155227661 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.421875 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7603346705436707 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07826567441225052 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8326908946037292 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.873498260974884 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7127083539962769 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 *jet/accuracy_2_of_3‚0.5418398976325989 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9976974725723267 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5951941609382629 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09673593193292618 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2752087116241455 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6528481841087341 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6644704937934875 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9545081853866577 jet/accuracy_4_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5720070600509644 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7268524169921875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7554196715354919 trainer/global_step‚15199 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3321627974510193 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0695570558309555 (particle/accuracy‚0.8516282439231873 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.59151691198349 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 *jet/accuracy_2_of_4‚0.7362229228019714 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8248299360275269 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6396127939224243 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.253654807806015 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/specificity‚0.9091889262199402 =&Purity/0FRt1SRqqt0²™¼AŒFBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8226568102836609 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2790645360946655 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2651425004005432 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3358817994594574 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975182414054871 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5751309990882874 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29530587792396545 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7094417214393616 +particle/sensitivity‚0.6411640644073486 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6965451240539551 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5916448831558228 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7017678618431091 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 *jet/accuracy_1_of_1‚0.3253680467605591 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6692708730697632 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46886876225471497 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2790645360946655 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05039868503808975 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5906200408935547 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6380884647369385 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5720070600509644 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7776075005531311 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7113857269287109 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5441666841506958 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0695570558309555 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6587918400764465 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9920614957809448 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 # _timestamp‚1.78452059185845e+09 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6573859453201294 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2703135907649994 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6327652335166931 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2949123978614807 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6162452101707458 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29035547375679016 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03979690745472908 *jet/accuracy_2_of_2‚0.1584746539592743 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7022863030433655 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29015353322029114 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_1‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.44794926047325134 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_3_of_4‚0.0943518728017807 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5599722862243652 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21066009998321533TLŠüC=š0½z 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 *jet/accuracy_1_of_1‚0.3253680467605591 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6692708730697632 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46886876225471497 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2790645360946655 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05039868503808975 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5906200408935547 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6380884647369385 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5720070600509644 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7776075005531311 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7113857269287109 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5441666841506958 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0695570558309555 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6587918400764465 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9920614957809448 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 # _timestamp‚1.78452059185845e+09 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6573859453201294 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2703135907649994 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6327652335166931 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2949123978614807 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6162452101707458 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29035547375679016 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03979690745472908 *jet/accuracy_2_of_2‚0.1584746539592743 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7022863030433655 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29015353322029114 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_1_of_1‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.44794926047325134 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_3_of_4‚0.0943518728017807 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5599722862243652 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21066009998321533 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1875000149011612 /particle/accuracy_3_of_3‚0.9994120597839355 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8402493596076965 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34041649103164673 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20431795716285706 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23857714235782623 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27648597955703735 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9923719763755798 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_3_of_4‚1 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚1 )loss/val_total_loss‚7.392778963674386 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34605470299720764 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6765676736831665 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19599315524101257 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8826034069061279 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6598713397979736 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 *jet/accuracy_1_of_4‚0.9750571250915527 6validation_average_jet_accuracy‚0.3654533624649048 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2192557156085968 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.752362072467804 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03853047266602516 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7084606885910034 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6172482371330261 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6568323373794556 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08083568513393402 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _step‚1253 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6844685673713684 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6187575459480286 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07880453765392303 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22092649340629578 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 epoch‚37 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 'particle/f_score‚0.6498445272445679 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5715315937995911 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31550484895706177 +jet/accuracy_3_of_3‚0.07661159336566925 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2005128413438797 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8267385363578796 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4911465048789978 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚13765.434516102 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26180902123451233 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2788272500038147 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6188724040985107 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26285091042518616 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_3‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5139511823654175 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6149253845214844 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7776075005531311 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4613560140132904 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04922683537006378 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08166537433862686 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5786425471305847 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2057950645685196 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5880967378616333 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2095503807067871 *jet/accuracy_1_of_3‚0.9104671478271484 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2501624524593353 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2907833158969879 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9447841644287109 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4873427748680115 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2516079246997833 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6695942878723145 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07806742936372757 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.645946204662323 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.909363865852356 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5882207155227661 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.421875 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7603346705436707 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07826567441225052 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8326908946037292 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.873498260974884 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7127083539962769 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 *jet/accuracy_2_of_3‚0.5418398976325989 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9976974725723267 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5951941609382629 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.09673593193292618 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2752087116241455 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6528481841087341 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6644704937934875 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9545081853866577 jet/accuracy_4_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5720070600509644 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7268524169921875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7554196715354919 trainer/global_step‚15199 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3321627974510193 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0695570558309555 (particle/accuracy‚0.8516282439231873 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.59151691198349 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 *jet/accuracy_2_of_4‚0.7362229228019714 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8248299360275269 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6396127939224243 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.253654807806015 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/specificity‚0.9091889262199402 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8226568102836609 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2790645360946655 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2651425004005432 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3358817994594574 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975182414054871 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5751309990882874 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29530587792396545 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7094417214393616 +particle/sensitivity‚0.6411640644073486 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6965451240539551 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5916448831558228 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7017678618431091å ]hñx›0‚(:í ”¿öÒÀ§éË& disk.nvme0n1.out‚90255.36083984375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0 disk.rbd0.in‚0 cpu‚25.68233471602226" disk.nvme1c1n1.out‚ 42256.46875( proc.memory.availableMB‚ 486.68359375 disk.md0.in‚0) disk./.usagePercent‚76.22130567447853% disk./.usageGB‚1170.1875953674316 disk.nvme1n1.out‚ 42256.46875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17 network.recv‚ 3.569816e+06 disk.md0.out‚ 662.48046875 disk.dm-0.in‚ 539.2421875" disk.dm-0.out‚116111.419921875# disk.dm-1.out‚15018.55908203125( disk.nvme0c0n1.out‚90255.36083984375$ memory_percent‚97.02951908111572# proc.memory.rssMB‚ 3024.74609375 network.sent‚ 2.4441401e+07 disk.nvme0n1.in‚ 309.37109375~aQžžœ0‚(:“ ”¿öÒÄž¯Ì gpu.0.gpu‚93.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.691' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.23033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03?B0‚(:ô £¿öÒµ¥³Ë( disk.nvme1c1n1.out‚42257.33251953125& disk.nvme1n1.out‚42257.33251953125 disk.rbd0.in‚0 network.sent‚ 2.4487231e+07" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 662.9453125" disk.dm-0.out‚116112.095703125 disk.dm-1.in‚0) proc.memory.percent‚18.46158504486084 network.recv‚ 3.573467e+06) disk./.usagePercent‚76.22133077035372% disk./.usageGB‚1170.1879806518555& disk.nvme0n1.out‚90256.47705078125 disk.md0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 3024.74609375 cpu‚25.682083839444324( disk.nvme0c0n1.out‚90256.47705078125 disk.nvme0n1.in‚ 309.37109375 disk.dm-0.in‚ 539.2421875! disk.dm-1.out‚15018.865234375# memory_percent‚97.0280647277832& proc.memory.availableMB‚ 486.921875 proc.cpu.threads‚17š«~¦žž0‚(:“ £¿öÒŸ¯úË gpu.0.gpu‚86.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚261.442' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.14733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô%þ€Ÿ0‚(:õ ²¿öÒ¬½ÀË disk.dm-0.in‚ 539.2421875' disk.nvme0c0n1.out‚90258.7666015625$ memory_percent‚97.02553749084473 cpu‚25.68186766651472 proc.cpu.threads‚17 network.sent‚ 2.4492071e+07 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 663.50390625# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 network.recv‚ 3.57696e+06) disk./.usagePercent‚76.22132580087349% disk.nvme0n1.out‚90258.7666015625 disk.md0.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚15018.935546875" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚42258.19580078125$ disk.dm-0.out‚116113.97509765625 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 487.3359375% disk./.usageGB‚1170.1879043579102 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚42258.19580078125ÁÉÛ½ž 0‚(:“ ²¿öÒÊì…Ì gpu.0.gpu‚82.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.082' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.02733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·nÂ4¡0/ _wandbruntime‚13807  _runtime‚13807–Ca¢Ê¢0Ä & lr-AdamW/pg2‚8.486374935109744e-05 trainer/global_step‚15249 % _timestamp‚1.7845206336245594e+09 _runtime‚13807.200769104 _step‚1254 & lr-AdamW/pg1‚8.486374935109744e-05 ½ÔsÏ£0É trainer/global_step‚15249 % _timestamp‚1.7845206336245594e+09 _runtime‚13807.200769104 _step‚1254 & lr-AdamW/pg1‚8.486374935109744e-05 & lr-AdamW/pg2‚8.486374935109744e-05æ ê_åZ4¤0/ _wandbruntime‚13807  _runtime‚13807§1š„¥0†P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2335030817055232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 _step‚1255 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123513857 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123513857 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26605521 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2321360953355728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2335030800015872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274436938 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2339055619113739 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2321360970395088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17694665758011 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17721393506304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63005343 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36399819 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2352752551985664 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2335030817055232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150923079 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150912936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚524806198544384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚524806076760064 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150912936 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123513859 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17721734930432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17820842068480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17695006319285 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2352752193522176 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2352752193522176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17721734930432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26605524 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2335030800015872 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150923079 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2352752551985664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143Ð Ç(¦0‹P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123513857 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26605521 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2321360953355728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2335030800015872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274436938 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2339055619113739 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2321360970395088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17694665758011 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17721393506304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63005343 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36399819 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2352752551985664 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2335030817055232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150923079 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150912936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚524806198544384 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚524806076760064 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150912936 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123513859 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17721734930432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17820842068480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17695006319285 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2352752193522176 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2352752193522176 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17721734930432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26605524 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2335030800015872 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150923079 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2352752551985664 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2335030817055232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 _step‚1255 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123513857 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2ç M×ß4§0/ _wandbruntime‚13807  _runtime‚13807v¥vS'¨0N bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17821956271616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚13807.817018467 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2352901936593920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150922592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507018536745984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2335179074518016 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123521650 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63009038 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36401783 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 % _timestamp‚1.7845206342412353e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17696114707649 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17722503605248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2321508384023200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63009329 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 274444242 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150932749 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274454401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150922592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26607258 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚507018518133760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚15249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123521650 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17722503605248 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2352901936593920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123521652 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150932749 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2339204158169463 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2339204515770209 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2321508401062560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 274454401 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2335179091557376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123521652 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2335179074518016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17722845036544 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17695774146263 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚13807.817018467 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2352901936593920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150922592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507018536745984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2335179074518016 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123521650 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63009038 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36401783 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 % _timestamp‚1.7845206342412353e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17696114707649 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17722503605248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2321508384023200 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63009329 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 274444242 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150932749 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274454401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150922592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26607258 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚507018518133760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚15249 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123521650 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17722503605248 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2352901936593920 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123521652 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150932749 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2339204158169463 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2339204515770209 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2321508401062560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 274454401 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2335179091557376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123521652 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2335179074518016 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17722845036544 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17695774146263 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2360221846194688 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2360221487731200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚13850.040140534 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2342444542162432 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17776962608128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2360221487731200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17776962608128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 275301095 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151397588 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63206615 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2360221487731200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17776962608128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 275301095 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151397588 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63206615 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2360221846194688 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2360221487731200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚13850.040140534 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2342444542162432 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17776962608128 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123911302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151417401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2342592799625216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63210844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2346631174756471 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2360371230802944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123911300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17778072707072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2328879934436160 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63210571 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275328703 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2342592816664576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 P@DeviceStatsMonitor.on_train_batn5㵡ch_end/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26693955 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚526555401943552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚508677531637760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚1260 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2328879917396800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17751257359671 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17778414138368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275318544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚15299 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2360371230802944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 $ _timestamp‚1.784520677079698e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17778072707072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36516886 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123911300 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151417401 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275318544 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2360370872332288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2342592816664576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2342592799625216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚508677550249984 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123911302 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526555288555008 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17877851693568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚13850.655508476 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3¾c“²'»0“N C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123911300 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17778072707072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2328879934436160 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63210571 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275328703 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2342592816664576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26693955 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚526555401943552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚508677531637760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚1260 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2328879917396800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17751257359671 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17778414138368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275318544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚15299 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2360371230802944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 $ _timestamp‚1.784520677079698e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17778072707072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36516886 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123911300 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151417401 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275318544 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2360370872332288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2342592816664576 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2342592799625216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚508677550249984 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123911302 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚526555288555008 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17877851693568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚13850.655508476 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123911302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151417401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2342592799625216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63210844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2346631174756471 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2360371230802944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85ì ²Í.i4¼0/ _wandbruntime‚13850  _runtime‚13850nJx½0‰ 3loss/SRqqt2/assignment_loss‚0.45222099702550095 /loss/FBt2/detection_loss‚0.3531970279912154 %loss/total_loss‚6.934927520847443 epoch‚38 trainer/global_step‚15299 _runtime‚13850.656104067 0loss/FRt2/assignment_loss‚0.6220072456451007 4loss/FBt1/assignment_loss‚0.00036090156013322524 /loss/FBt1/detection_loss‚0.5476963463633013 3loss/SRqqt1/assignment_loss‚0.35764148943596397 3loss/FBt2/assignment_loss‚7.357365013357745e-05 1loss/SRqqt1/detection_loss‚0.6257648643371813 1loss/SRqqt2/detection_loss‚0.6169665277923707 $ _timestamp‚1.784520677080768e+09 0loss/FRt1/assignment_loss‚1.4828524066477407 -loss/FRt1/detection_loss‚0.46629451349329 /loss/FRt2/detection_loss‚1.4098516269055106 _step‚1261•ý5¾0Ž %loss/total_loss‚6.934927520847443 epoch‚38 trainer/global_step‚15299 _runtime‚13850.656104067 0loss/FRt2/assignment_loss‚0.6220072456451007 4loss/FBt1/assignment_loss‚0.00036090156013322524 /loss/FBt1/detection_loss‚0.5476963463633013 3loss/SRqqt1/assignment_loss‚0.35764148943596397 3loss/FBt2/assignment_loss‚7.357365013357745e-05 1loss/SRqqt1/detection_loss‚0.6257648643371813 1loss/SRqqt2/detection_loss‚0.6169665277923707 $ _timestamp‚1.784520677080768e+09 0loss/FRt1/assignment_loss‚1.4828524066477407 -loss/FRt1/detection_loss‚0.46629451349329 /loss/FRt2/detection_loss‚1.4098516269055106 _step‚1261 3loss/SRqqt2/assignment_loss‚0.45222099702550095 /loss/FBt2/detection_loss‚0.3531970279912154í 2b|¿0‚(:ö î¿öÒ•ÙøË disk.dm-1.in‚0" disk.dm-1.out‚15019.3369140625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚90299.01416015625 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 664.73828125# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 cpu‚25.680953450232426( disk.nvme1c1n1.out‚42261.76904296875% disk./.usageGB‚1168.4714241027832( disk.nvme0c0n1.out‚90299.01416015625 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875! disk.dm-0.out‚116154.69921875$ memory_percent‚97.02620506286621' proc.memory.availableMB‚ 487.2265625 network.sent‚ 2.4579934e+07) disk./.usagePercent‚76.10952119216957 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme1n1.out‚42261.76904296875 proc.cpu.threads‚17" disk.nvme1c1n1.in‚ 229.87109375 network.recv‚ 3.59062e+06ï˸žÀ0‚(:“ î¿öÒ³ì¹Ì gpu.0.gpu‚40.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.284' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.76133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G¢_Á0‚(:ô ý¿öÒ“ïáË network.recv‚ 3.594332e+06" disk.nvme1c1n1.in‚ 229.87109375% disk.nvme1n1.out‚42262.3720703125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚92185.6826171875) disk./.usagePercent‚76.22425804268632% disk./.usageGB‚1170.2329216003418 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0" disk.nvme0c0n1.in‚ 309.37109375& proc.memory.availableMB‚ 487.234375 disk.md0.out‚ 665.140625 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚118040.99658203125# disk.dm-1.out‚15019.45947265625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 network.sent‚ 2.4630609e+07' disk.nvme1c1n1.out‚42262.3720703125 disk.dm-1.in‚0% disk.nvme0n1.out‚92185.6826171875$ memory_percent‚97.02615737915039 cpu‚25.680740750822054 proc.cpu.threads‚17³`“Â0‚(:ˆ ý¿öÒñˆ¥Ì gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚290.538' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.846 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó-‚a~Ã0‚(:ó ŒÀöÒõú§Ë" disk.nvme0c0n1.in‚ 309.37109375$ memory_percent‚97.02615737915039# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17 disk.rbd0.out‚0( disk.nvme0c0n1.out‚92189.40283203125) disk./.usagePercent‚76.22422897122695 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚92189.40283203125 network.sent‚ 2.463548e+07# disk.dm-0.out‚118044.8017578125 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1c1n1.out‚42262.896484375 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚42262.896484375& proc.memory.availableMB‚ 487.234375 network.recv‚ 3.598042e+06% disk./.usageGB‚1170.2324752807617 disk.md0.out‚ 665.3203125# disk.dm-1.out‚15019.48291015625) proc.memory.percent‚18.46158504486084 cpu‚25.680529564624774ߔǒÄ0‚(:‡ ŒÀöÒ³¦íË gpu.0.gpu‚13.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.995' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@"n4Å0/ _wandbruntime‚13892  _runtime‚13892ŽJ§zÊÆ0Ä _step‚1262 & lr-AdamW/pg1‚8.149433505158972e-05 & lr-AdamW/pg2‚8.149433505158972e-05 trainer/global_step‚15349 % _timestamp‚1.7845207193321111e+09 _runtime‚13892.907717398É©þÏÇ0É & lr-AdamW/pg2‚8.149433505158972e-05 trainer/global_step‚15349 % _timestamp‚1.7845207193321111e+09 _runtime‚13892.907717398 _step‚1262 & lr-AdamW/pg1‚8.149433505158972e-05î ÀfÝ4È0/ _wandbruntime‚13892  _runtime‚13892–‚6Î (É0…P JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17932640088064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17832531709952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36629475 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2353910009888389 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784520719333013e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124293157 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2367691140403712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63408399 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151882240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2336104020102928 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17805632184827 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151892383 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17832873134080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151882240 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2349858250230272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17832873134080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26778924 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2349858250230272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17832531709952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17932539013120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2336104037142288 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17805972746101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36629194 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124293159 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2367690781940224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26778921 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2367691140403712 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2367690781940224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276185542 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276175397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚528236023571968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2349858267269632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151892383 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _step‚1263 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124293159 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124293157 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚13892.908465699 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2349858267269632Ù*¿Y+ Ê0ŠP _step‚1263 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124293159 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124293157 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚13892.908465699 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2349858267269632 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17932640088064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_ˆë%åpool.freed‚17832531709952 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36629475 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2353910009888389 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784520719333013e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124293157 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2367691140403712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63408399 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151882240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2336104020102928 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17805632184827 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151892383 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17832873134080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151882240 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2349858250230272 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17832873134080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26778924 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2349858250230272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17832531709952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17932539013120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2336104037142288 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17805972746101 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36629194 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124293159 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2367690781940224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26778921 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2367691140403712 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2367690781940224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276185542 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276175397 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚528236023571968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2349858267269632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151892383 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272ï <1‰°4Ë0/ _wandbruntime‚13893  _runtime‚13893bÅ?'Ì0N WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124300950 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17933753822208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107097088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17833983240192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151891896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2354058548944225 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124300952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17833983240192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124300952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚15349 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36631746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2336251467809760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26780658 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17933665337344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845207199479587e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2350006541771776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚528270210479104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88484864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36631465 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2354058191343479 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151902053 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2367840166541312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17833641808896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17806740573079 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26780655 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510336545141760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151891896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17833641808896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚13893.523753352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63412120 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2367840525011968 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2350006524732416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276203005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2350006524732416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276203005 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124300950 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17933753822208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107097088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17833983240192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151891896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2354058548944225 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124300952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17833983240192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124300952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚15349 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36631746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2336251467809760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26780658 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17933665337344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845207199479587e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2350006541771776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚528270210479104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88484864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36631465 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2354058191343479 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151902053 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2367840166541312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17833641808896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17806740573079 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26780655 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510336545141760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151891896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚17833641808896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚13893.523753352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63412120 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2367840525011968 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2375160076149248 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2357271975337472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17888100811776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚529950883271680 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2343475553476528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚15399 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚511962396987904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2343475570515888 _runtime‚13935.560679247 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2375160434612736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124682809 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152377035 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36744840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344÷ø(Ü0‹P QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63610464 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17861455959509 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152366892 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2357271992376832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 trainer/global_step‚15399 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚511962396987904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2343475570515888 _runtime‚13935.560679247 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2375160434612736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124682809 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152377035 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36744840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2357271992376832 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124682807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26865621 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2375160076149248 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2357271975337472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17888100811776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚529950883271680 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2343475553476528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277077307 _step‚1268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152376548 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚17889552342016 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2375309460750336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2343622984144000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63614136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚511995558645760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17862223786487 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚529985178879488 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2357420266878976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26867355 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 277067148 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63614422 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17989498456576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17889210910720 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2375309819220992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2375309819220992 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277077307 _step‚1268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152376548 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚1788955v,¤î$2342016 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2375309460750336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2343622984144000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63614136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚511995558645760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17862223786487 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚529985178879488 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2357420266878976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26867355 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 277067148 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63614422 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17989498456576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚17889210910720 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2375309819220992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2375309819220992 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2364685717484032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 _step‚1271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2364685700444672 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2382629728821760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513621410491904 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152861687 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2382629370358272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152851544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2364685700444672 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 277924001 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26952321 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125072459 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17944011337728 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63811077 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152861687 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452ST^M!ì0‹P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17944011337728 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63811077 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152861687 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2364685717484032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 _step‚1271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2364685700444672 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2382629728821760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513621410491904 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152861687 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2382629370358272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152851544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2364685700444672 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 277924001 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26952321 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125072459 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528÷ “’ÊL4í0/ _wandbruntime‚13978  _runtime‚13978^ÏlU'î0N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513654590761984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2368912582118241 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2350994534556960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513654572149760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125080252 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚302 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2382779113430016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17945121443840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1272 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277951609 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63815322 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2350994517517600 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2364833974946816 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125080252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2364833974946816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125080250 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63815017 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63815017 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513654590761984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2368912582118241 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2350994534556960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513654572149760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125080252 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚302 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2382779113430016 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚17945121443840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1272 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277951609 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63815322 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2350994517517600 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2364833974946816 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125080252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2364833974946816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125080250 QDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18100126618112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18100029737472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2376191059649413 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36973371 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125462107 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17999239015424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2372099425551872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17999580439552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36973093 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533380435121152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278808448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17972081825051 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2358218620223728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2372099442591232 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2372099442591232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _runtime‚14020.660973078 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2390098664567296 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2376190702048779 F@DeviceStatsMonì^ß+itor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64012114 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _step‚1275 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2390099023030784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125462109 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2390099023030784 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125462109 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27039021 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125462107 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17999239015424 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153346339 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533380550614016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2390098664567296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153336196 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27039024 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17972422386325 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚515280405383680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17999580439552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64012395 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 trainer/global_step‚15499“@ ´(þ0ŠP GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚17972422386325 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚515280405383680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚17999580439552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64012395 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 trainer/global_step‚15499 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18100126618112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18100029737472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2376191059649413 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36973371 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125462107 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚17999239015424 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2372099425551872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚17999580439552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36973093 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533380435121152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278808448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚17972081825051 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2358218620223728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2372099442591232 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2372099442591232 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _runtime‚14020.660973078 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2390098664567296 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2376190702048779 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64012114 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _step‚1275 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2390099023030784 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125462109 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2390099023030784 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125462109 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27039021 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125462107 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚17999239015424 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153346339 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533380550614016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2390098664567296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153336196 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27039024û uÞìÛ4ÿ0/ _wandbruntime‚14021  _runtime‚14021¨qR'€1N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18000690545664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64016145 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36975390 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18000349114368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533414851468800 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2376339241104503 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2372247717093376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17973190213303 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125469902 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125469902 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17973530774689 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18000690545664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2372247700054016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _step‚1276 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278825911 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2372247717093376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125469900 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27040755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153345852 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64016438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚515313585653760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278815752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153356009 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64016145 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36975390 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18000349114368 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533414851468800 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2376339241104503 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2372247717093376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚17973190213303 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125469902 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125469902 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚17973530774689 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18000690545664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2372247700054016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _step‚1276 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278825911 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2372247717093376 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125469900 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27040755 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153345852 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64016438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚515313585653760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278815752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153356009 F)ò†1‚(:’ “ÁöÒœÍÈÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚286.82' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.60666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸµÊŇ1‚(:ô ¢ÁöÒ»ÄÐË disk.md0.out‚ 668.4453125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚92857.66748046875( disk.nvme1c1n1.out‚42498.40478515625 proc.cpu.threads‚17 disk.nvme1n1.in‚ 229.87109375# disk.dm-0.out‚118933.1298828125& disk.nvme0n1.out‚92857.66748046875" disk.nvme1c1n1.in‚ 229.87109375' proc.memory.availableMB‚ 486.5078125 cpu‚25.678459564489266& disk.nvme1n1.out‚42498.40478515625 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚97.03059196472168# proc.memory.rssMB‚ 3024.74609375 network.sent‚ 2.4912724e+07 network.recv‚ 3.634507e+06) disk./.usagePercent‚76.22124181665751% disk./.usageGB‚1170.1866149902344 disk.dm-1.out‚ 15028.078125" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375) proc.memory.percent‚18.46158504486084 disk.md0.in‚0Š6/žˆ1‚(:“ ¢ÁöÒø™¤Ì gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚277.547' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.51566666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GreЄ‰1‚(:ù ±ÁöÒɧÞË disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17 network.recv‚ 3.637703e+06) disk./.usagePercent‚76.22122069636652 disk.nvme1n1.in‚ 229.87109375# disk.dm-0.out‚118933.6728515625( proc.memory.availableMB‚ 486.12109375 cpu‚25.678286679233352" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚42499.0791015625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚92858.60107421875 disk.md0.in‚0$ memory_percent‚97.03295230865479) proc.memory.percent‚18.46158504486084 network.sent‚ 2.4919477e+07& disk.nvme0n1.out‚92858.60107421875% disk.nvme1n1.out‚42499.0791015625 disk.md0.out‚ 668.85546875 disk.dm-0.in‚ 539.2421875% disk./.usageGB‚1170.1862907409668 disk.dm-1.in‚0# disk.dm-1.out‚15028.27587890625íöî…Š1‚(:’ ±ÁöÒœö­Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.955' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.318333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ6š4‹1/ _wandbruntime‚14063  _runtime‚14063›-!5ÊŒ1Ä % _timestamp‚1.7845208896693418e+09 _runtime‚14063.244933737 _step‚1278 & lr-AdamW/pg1‚7.493034868715036e-05 & lr-AdamW/pg2‚7.493034868715036e-05 trainer/global_step‚15549.Š£Ï1É & lr-AdamW/pg1‚7.493034868715036e-05 & lr-AdamW/pg2‚7.493034868715036e-05 trainer/global_step‚15549 % _timestamp‚1.7845208896693418e+09 _runtime‚14063.244933737 _step‚1278þ £(Î4Ž1/ _wandbruntime‚14063  _runtime‚14063ÜEEæ (1†P SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 279672605 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18156002872320 _step‚1279 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18055149541376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18054808117248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153820848 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64212533 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125851759 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279682750 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚516939437499904 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2379513150659072 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2397568317239808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2379513167698432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚15549 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153830991 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18054808117248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64212840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚304 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚535095440372224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 117852160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153820848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37087116 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2383617718635787 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2365590153597328 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2379513167698432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 136464384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2397567958776320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153830991 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18155885020160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18055149541376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚307 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2365590170636688 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2379513150659072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2383618076236421 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27125721 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125851757 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37086812 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27125724 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18027565038459 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125851759"1×4(1‹P VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2383617718635787 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2365590153597328 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2379513167698432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 136464384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2397567958776320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153830991 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18155885020160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18055149541376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚307 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2365590170636688 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2379513150659072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2383618076236421 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27125721 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125851757 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37086812 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27125724 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18027565038459 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125851759 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 279672605 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18156002872320 _step‚1279 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18055149541376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18054808117248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153820848 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64212533 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125851759 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279682750 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚516939437499904 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2379513150659072 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2397568317239808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2379513167698432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚15549 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153830991 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18054808117248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64212840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚304 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚535095440372224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 117852160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153820848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37087116 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ÿ ¿Uµ'4‘1/ _wandbruntime‚14063  _runtime‚14063-éY ’1N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18055918216192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18056259647488 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2379661442200576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2383766257691511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 279690054 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18056259647488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.fr¯#dd¼eed‚516972599157760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2365737584264800 % _timestamp‚1.7845208902817888e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64216892 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2397717701848064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18055918216192 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 279700213 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125859550 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2397717701848064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 279690054 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64216605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279700213 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚535129608371712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18028673426711 trainer/global_step‚15549 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153840661 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2379661425161216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153830504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18157106087424 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2365737601304160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚516972617769984 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125859550 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27127455 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0äc'“1”N F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18029013988097 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2379661442200576 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2383766615292257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚14063.857560383 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125859552 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚535129723857408 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153840661 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27127458 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 279700213 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125859550 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2397717701848064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 279690054 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64216605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279700213 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚535129608371712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18028673426711 trainer/global_step‚15549 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153840661 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2379661425161216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153830504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18157106087424 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2365737601304160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚516972617769984 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125859550 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27127455 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18055918216192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18056259647488 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2379661442200576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2383766257691511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 279690054 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18056259647488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚516972599157760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2365737584264800 % _timestamp‚1.7845208902817888e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64216892 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2397717701848064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18055918216192 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154305500 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18110718643200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18110718643200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18110377219072 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2405037611448832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2386926892805632 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154305500 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2391045092823429 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2386926892805632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18083388813141 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126241409 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚518598432391680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2405037252985344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126241409 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37202335 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300î£Þ(¢1ŠP ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2386926875766272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126241407 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126241407 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27212421 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2405037252985344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚536810298001920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27212424 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2372961686970928 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18083048251867 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37202035 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 $ _timestamp‚1.784520931953627e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154305500 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18110718643200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18110718643200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18110377219072 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2405037611448832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2386926892805632 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154305500 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2391045092823429 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2386926892805632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18083388813141 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126241409 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚518598432391680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2405037252985344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126241409 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37202335 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚300 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64414456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2391044735222795 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚15599 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ƒ š¸ö4£1/ _wandbruntime‚14106  _runtime‚14106.Í€Ç'¤1N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280574515 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64418493 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2387075167307776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126249202 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126249200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64418800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚536844592851968 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126249200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7845209325661902e+09 _step‚1284 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280564356 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126249202 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280564356 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37204338 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2391193274278519 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2405186996057088 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2405186637586432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154325313 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154315156 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18111487318016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _runtime‚14106.141954742 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2387075150268416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27214158 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37204642 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2387075150268416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154325313 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2405186996057088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18084156640119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18212852121600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚536844464783360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2373109134677760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154315156 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚518631631273984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18111828749312 trainer/global_step‚15599 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280574515 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚518631612661760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0Ü.'¥1•N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2405186996057088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18084156640119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18212852121600 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚536844464783360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2373109134677760 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154315156 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚518631631273984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18111828749312 trainer/global_step‚15599 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280574515 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚518631612661760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280574515 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64418493 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2387075167307776 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126249202 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126249200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64418800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚536844592851968 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126249200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7845209325661902e+09 _step‚1284 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280564356 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126249202 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280564356 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37204338 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2391193274278519 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2405186996057088 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2405186637586432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154325313 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154315156 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18111487318016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _runtime‚14106.141954742 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2387075150268416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27214158 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37204642 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2387075150268416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154325313 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528„ ljÑA4¦1/ _wandbruntime‚14106  _runtime‚14106bǬ§1ˆ 3loss/SRqqt2/assignment_loss‚0.43125276968874493 /loss/FRt1/detection_loss‚0.4532084558131485 .loss/FRt2/detection_loss‚1.527964009161803 1loss/SRqqt1/detection_loss‚0.5938906314163852 epoch‚38 0loss/FRt1/assignment_loss‚1.5214539320963734 2loss/SRqqt1/assignment_loss‚0.3378995801925048 3loss/FBt1/assignment_loss‚8.188568786892144e-05 4loss/FBt2/assignment_loss‚1.2512241111650724e-05 %loss/total_loss‚6.816284083393603 _step‚1285 1loss/SRqqt2/detection_loss‚0.6066422980095793 _runtime‚14106.142446073 0loss/FRt2/assignment_loss‚0.6033521160588052 -loss/FBt1/detection_loss‚0.44826303083657 /loss/FBt2/detection_loss‚0.2922628621907077 trainer/global_step‚15599 % _timestamp‚1.7845209325671685e+097›\Z¨1 0loss/FRt2/assignment_loss‚0.6033521160588052 -loss/FBt1/detection_loss‚0.44826303083657 /loss/FBt2/detection_loss‚0.2922628621907077 trainer/global_step‚15599 % _timestamp‚1.7845209325671685e+09 3loss/SRqqt2/assignment_loss‚0.43125276968874493 /loss/FRt1/detection_loss‚0.4532084558131485 .loss/FRt2/detection_loss‚1.527964009161803 1loss/SRqqt1/detection_loss‚0.5938906314163852 epoch‚38 0loss/FRt1/assignment_loss‚1.5214539320963734 2loss/SRqqt1/assignment_loss‚0.3378995801925048 3loss/FBt1/assignment_loss‚8.188568786892144e-05 4loss/FBt2/assignment_loss‚1.2512241111650724e-05 %loss/total_loss‚6.816284083393603 _step‚1285 1loss/SRqqt2/detection_loss‚0.6066422980095793 _runtime‚14106.142446073… D/K$©1‚(:ò íÁöÒç«ßË disk.mdEe`LY0.out‚ 670.34375! disk.dm-1.out‚15031.861328125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚42505.14111328125& disk.nvme1n1.out‚42505.14111328125% disk./.usageGB‚1167.4698066711426 disk.md0.in‚0$ disk.dm-0.out‚118942.03369140625 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚92867.6728515625% disk.nvme0n1.out‚92867.6728515625$ memory_percent‚97.03564643859863 network.sent‚ 2.5007586e+07 network.recv‚ 3.651951e+06 disk.dm-0.in‚ 539.2421875 disk.nvme0n1.in‚ 309.37109375' proc.memory.availableMB‚ 485.6796875# proc.memory.rssMB‚ 3024.74609375 cpu‚25.67942262840907 proc.cpu.threads‚17( disk./.usagePercent‚76.0442798678484 disk.nvme1n1.in‚ 229.87109375) proc.memory.percent‚18.46158504486084“ª1‚(:ˆ íÁöÒ£Ü§Ì gpu.0.gpu‚71.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.345' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.115 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çPFû4«1/ _wandbruntime‚14122  _runtime‚14122_¹ËÕ„=¬1þz 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27340781688690186 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9132363200187683 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4861111044883728 +particle/specificity‚0.9095372557640076 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4488159716129303 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6382931470870972 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7042314410209656 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6702132225036621 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 )loss/val_total_loss‚7.416491543363837 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07219711691141129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 trainer/global_step‚15599 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.23666667938232422 +jet/accuracy_3_of_4‚0.10072942823171616 epoch‚38 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6437715291976929 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5902107357978821 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 particle/accuracy_5_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2775227427482605 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3382028043270111 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5589613914489746 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2922123372554779 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7618993520736694 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8685259222984314 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6365296244621277 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5691941380500793 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8514655828475952 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04986618831753731 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7630208730697632 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9553312659263611 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 /particle/accuracy_3_of_3‚0.9994120597839355 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.270331472158432 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29892000555992126 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.061825137585401535 % _timestamp‚1.7845209493705795e+09 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5892372727394104 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7819161415100098 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03890830650925636 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07606060057878494 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 particle/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 particle/accuracy_1_of_3‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07268577069044113 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.712096095085144 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.32980093359947205 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6472746729850769 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3019182085990906 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3034805655479431 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11057574301958084 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.29127514362335205 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6380426287651062 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 5validation_average_jet_accuracy‚0.368701308965683 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7021504640579224 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2925752103328705 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 particle/accuracy_1_of_5‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.592549204826355 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6971414685249329 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.570543110370636 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6195833086967468 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7572415471076965 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08701150119304657 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.044593408703804016 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6784125566482544 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8622449040412903 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07682077586650848 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25290921330451965 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.32140374183654785 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.061825137585401535 jet/accuracy_4_of_5‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4621391296386719 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6999801397323608 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 (jet/accuracy_2_of_4‚0.71631920337677 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6171284914016724 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20258471369743347 'particle/f_score‚0.6488857865333557 _runtime‚14122.946672866 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6153748631477356 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3397045135498047 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.15786820650100708 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6252389550209045 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6115842461585999 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7618993520736694 +particle/sensitivity‚0.6391404271125793 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6702149510383606 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23029205203056335 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2202669382095337 jet/accuracy_5_of_5‚0 /particle/accuracy_4_of_4‚0.9514349699020386 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.497177392244339 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26251527667045593 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5959698557853699 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34812164306640625 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2928639352321625 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5091715455055237 *jet/accuracy_1_of_2‚0.7562339901924133 +jet/accuracy_3_of_3‚0.07603008300065994 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2130301147699356 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5897259712219238 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26147666573524475 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6802402138710022 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5731092691421509 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4991675019264221 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9970559477806091 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (jet/accuracy_1_of_4‚0.95341956615448 jet/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9923719763755798 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5939353108406067 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6158997416496277 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.892684817314148 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8682917356491089 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6644797921180725 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6539366841316223 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28331732749938965 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8617347478866577 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6767821311950684 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2162703573703766 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.570543110370636 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8562925457954407 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5572916269302368 _step‚1286 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2922123372554779 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6986060738563538 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8603552579879761 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45216384530067444 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6145365834236145 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9920614957809448 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2268972098827362 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5751132965087891 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9973881840705872 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3050553798675537 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27377423644065857 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30332815647125244 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.056768711656332016 *jet/accuracy_1_of_3‚0.9151322245597839 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2051117718219757 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_3‚0.5461490750312805 jet/accuracy_1_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7289669513702393 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34448814392089844 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20131123065948486 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7163833379745483 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192•d3.9=­1ƒ{ 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8603552579879761 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45216384530067444 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6145365834236145 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9920614957809448 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2268972098827362 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5751132965087891 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9973881840705872 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3050553798675537 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_5‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27377423644065857 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30332815647125244 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.056768711656332016 *jet/accuracy_1_of_3‚0.9151322245597839 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2051117718219757 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_2_of_3‚0.5461490750312805 jet/accuracy_1_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9972478747367859 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7289669513702393 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34448814392089844 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20131123065948486 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7163833379745483 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27340781688690186 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.997602105140686 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9132363200187683 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4861111044883728 +particle/specificity‚0.9095372557640076 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4488159716129303 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6382931470870972 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7042314410209656 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6702132225036621 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 )loss/val_total_loss‚7.416491543363837 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07219711691141129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 trainer/global_step‚15599 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.23666667938232422 +jet/accuracy_3_of_4‚0.10072942823171616 epoch‚38 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6437715291976929 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5902107357978821 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 particle/accuracy_5_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2775227427482605 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3382028043270111 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5589613914489746 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2922123372554779 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7618993520736694 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8685259222984314 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6365296244621277 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5691941380500793 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8514655828475952 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04986618831753731 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7630208730697632 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9553312659263611 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 /particle/accuracy_3_of_3‚0.9994120597839355 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.270331472158432 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29892000555992126 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.061825137585401535 % _timestamp‚1.7845209493705795e+09 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5892372727394104 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9966611266136169 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7819161415100098 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03890830650925636 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07606060057878494 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 particle/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 particle/accuracy_1_of_3‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07268577069044113 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.712096095085144 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.32980093359947205 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6472746729850769 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3019182085990906 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3034805655479431 +jet/accuracy_4_of_4‚0.01670061983168125 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11057574301958084 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.29127514362335205 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6380426287651062 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 5validation_average_jet_accuracy‚0.368701308965683 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7021504640579224 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2925752103328705 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 particle/accuracy_1_of_5‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.592549204826355 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6971414685249329 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.570543110370636 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6195833086967468 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7572415471076965 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08701150119304657 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.044593408703804016 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6784125566482544 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8622449040412903 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07682077586650848 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25290921330451965 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.32140374183654785 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.061825137585401535 jet/accuracy_4_of_5‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4621391296386719 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6999801397323608 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 (jet/accuracy_2_of_4‚0.71631920337677 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6171284914016724 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20258471369743347 'particle/f_score‚0.6488857865333557 _runtime‚14122.946672866 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6153748631477356 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3397045135498047 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.15786820650100708 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6252389550209045 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6115842461585999 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7618993520736694 +particle/sensitivity‚0.6391404271125793 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6702149510383606 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23029205203056335 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2202669382095337 jet/accuracy_5_of_5‚0 /particle/accuracy_4_of_4‚0.9514349699020386 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.497177392244339 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26251527667045593 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5959698557853699 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34812164306640625 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2928639352321625 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5091715455055237 *jet/accuracy_1_of_2‚0.7562339901924133 +jet/accuracy_3_of_3‚0.07603008300065994 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2130301147699356 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5897259712219238 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26147666573524475 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6802402138710022 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5731092691421509 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_2‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4991675019264221 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9970559477806091 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (jet/accuracy_1_of_4‚0.95341956615448 jet/accuracy_2_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9923719763755798 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5939353108406067 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6158997416496277 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.892684817314148 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8682917356491089 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6644797921180725 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6539366841316223 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28331732749938965 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8617347478866577 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6767821311950684 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2162703573703766 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.570543110370636 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8562925457954407 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5572916269302368 _step‚1286 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.@=(P2922123372554779 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6986060738563538† Àf;k®1‚(:ö üÁöÒñ¶ÑË) proc.memory.percent‚18.46158504486084 cpu‚25.68147720921664 proc.cpu.threads‚17 network.sent‚ 2.5069043e+07 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚118953.61181640625 disk.dm-1.in‚0 disk.md0.out‚ 670.48828125) disk./.usagePercent‚76.04428111021846 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚42506.95849609375 disk.md0.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 network.recv‚ 3.655404e+06& disk.nvme1n1.out‚42506.95849609375" disk.dm-1.out‚15031.9873046875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚92877.8876953125$ memory_percent‚97.03507423400879' proc.memory.availableMB‚ 485.7734375# proc.memory.rssMB‚ 3024.74609375$ disk./.usageGB‚1167.469825744629% disk.nvme0n1.out‚92877.8876953125 disk.nvme1n1.in‚ 229.87109375’ ¾_“¯1‚(:ˆ üÁöÒÛÅÌ gpu.0.gpu‚68.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.098' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.366 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀœÃw°1‚(:ì ‹ÂöÒÓͲË) proc.memory.percent‚18.46158504486084 cpu‚25.681939090781484 proc.cpu.threads‚17 network.sent‚ 2.5113096e+07 network.recv‚ 3.659246e+06) disk./.usagePercent‚76.04428235258852 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0% disk./.usageGB‚1167.4698448181152" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚42507.36376953125& disk.nvme1n1.out‚42507.36376953125 disk.md0.out‚670.75 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375" disk.nvme0n1.out‚ 92878.7578125 disk.nvme1n1.in‚ 229.87109375$ disk.dm-0.out‚118954.24072265625" disk.dm-1.out‚15032.0419921875$ memory_percent‚97.03469276428223' proc.memory.availableMB‚ 485.8359375$ disk.nvme0c0n1.out‚ 92878.7578125 disk.dm-0.in‚ 539.2421875# proc.memory.rssMB‚ 3024.74609375Ò˜“±1‚(:ˆ ‹ÂöÒªö†Ì gpu.0.gpu‚56.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚278.634' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹• ²1‚(:ô šÂöÒ—¢ºË" disk.nvme0c0n1.in‚ 309.37109375 disk.md0.out‚ 670.91015625 network.sent‚ 2.5119134e+07( disk.nvme1c1n1.out‚42507.93505859375 disk.md0.in‚0& proc.memory.availableMB‚ 486.015625# proc.memory.rssMB‚ 3024.74609375 disk.nvme0n1.in‚ 309.37109375% disk.nvme0n1.out‚92880.3251953125 disk.dm-1.in‚0 disk.rbd0.out‚0) proc.memory.percent‚18.46158504486084 cpu‚25.6823459481942 proc.cpu.threads‚17$ disk./.usageGB‚1167.469913482666 disk.dm-0.in‚ 539.2421875' disk.nvme0c0n1.out‚92880.3251953125" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚42507.93505859375$ memory_percent‚97.03359603881836 network.recv‚ 3.662939e+06) disk./.usagePercent‚76.04428682512074$ disk.dm-0.out‚118955.77783203125" disk.dm-1.out‚15032.2783203125 disk.rbd0.in‚0!œÂb“³1‚(:ˆ šÂöÒ㱘Ì gpu.0.gpu‚88.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚271.314' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚90.438 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G•Ý4´1/ _wandbruntime‚14165  _runtime‚14165€Ü˜Êµ1Ä & lr-AdamW/pg2‚7.173746296545407e-05 trainer/global_step‚15649 % _timestamp‚1.7845209920469298e+09 _runtime‚14165.622540003 _step‚1287 & lr-AdamW/pg1‚7.173746296545407e-05tuö϶1É _runtime‚14165.622540003 _step‚1287 & lr-AdamW/pg1‚7.173746296545407e-05 & lr-AdamW/pg2‚7.173746296545407e-05 trainer/global_step‚15649 % _timestamp‚1.7845209920469298e+09‡ ðU§X4·1/ _wandbruntime‚14165  _runtime‚14165=žÚ6 (¸1†P UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18159175875446 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281635656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2382254476352752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154881789 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27303237 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2400413669267558 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154871646 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2396282944749056 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154871646 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2414469208905728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27303240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281625511 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126753865 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2414469567369216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37354968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚520280043170304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18288471665152 _step‚1288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚538568514835456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2382254493392112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18186605580800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64657921 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18186605580800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚520280024558080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18288366395904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2414469208905728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚538568390953984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2414469567369216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154881789 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0NÂŽñ(¹1‹P N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18186605580800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64657921 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18186605580800 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚520280024558080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18288366395904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2414469208905728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚538568390953984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2414469567369216 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154881789 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18159175875446 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281635656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2382254476352752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154881789 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27303237 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2400413669267558 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154871646 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2396282944749056 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154871646 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2414469208905728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27303240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281625511 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126753865 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2414469567369216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37354968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚520280043170304 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18288471665152 _step‚1288 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚538568514835456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2382254493392112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452ˆ )4Õ 4º1/ _wandbruntime‚14166  _runtime‚14166ï!’üÅ»1N FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18159943702424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27304974 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126761660 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18187715686912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚1289 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18187374255616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281653119 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18160284263810 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚538602691228160 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2400561850722648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚15649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281642960 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126761658 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154891459 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2400562208323394 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126761658 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154881302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2382401924059584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚538602813005312 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64661941 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚520313204828160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatse‘ŽQ Monitor.on_train_batch_end/active.small_pool.allocated‚ 154891459 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18289589564928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344#ÛÊD'¼1•N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚538602691228160 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2400561850722648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚15649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281642960 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126761658 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154891459 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2400562208323394 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126761658 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154881302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2382401924059584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚538602813005312 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64661941 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚520313204828160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154891459 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18289589564928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18159943702424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27304974 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126761660 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18187715686912 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚1289 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18187374255616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281653119 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18160284263810 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386‰ äþ4½1/ _wandbruntime‚14166  _runtime‚14166‡qJÕ¾1Š % _timestamp‚1.7845209926614306e+09 _step‚1290 3loss/SRqqt2/assignment_loss‚0.45229502220170537 /loss/FBt1/detection_loss‚0.4460754802201678 epoch‚39 %loss/total_loss‚7.167039164052859 0loss/FRt2/assignment_loss‚0.6637614507164689 3loss/FBt1/assignment_loss‚0.0022794394834785947 0loss/FRt1/detection_loss‚0.49619784977406917 /loss/FRt2/detection_loss‚1.4513724744535186 1loss/SRqqt1/detection_loss‚0.6864269017474726 1loss/SRqqt2/detection_loss‚0.5731407583558391 /loss/FBt2/detection_loss‚0.4480898860237343 0loss/FRt1/assignment_loss‚1.5177196060825964 1loss/SRqqt1/assignment_loss‚0.429647761343261 trainer/global_step‚15649 _runtime‚14166.240006026 3loss/FBt2/assignment_loss‚3.253365054747542e-05&âÛ=¿1 3loss/FBt2/assignment_loss‚3.253365054747542e-05 % _timestamp‚1.7845209926614306e+09 _step‚1290 3loss/SRqqt2/assignment_loss‚0.45229502220170537 /loss/FBt1/detection_loss‚0.4460754802201678 epoch‚39 %loss/total_loss‚7.167039164052859 0loss/FRt2/assignment_loss‚0.6637614507164689 3loss/FBt1/assignment_loss‚0.0022794394834785947 0loss/FRt1/detection_loss‚0.49619784977406917 /loss/FRt2/detection_loss‚1.4513724744535186 1loss/SRqqt1/detection_loss‚0.6864269017474726 1loss/SRqqt2/detection_loss‚0.5731407583558391 /loss/FBt2/detection_loss‚0.4480898860237343 0loss/FRt1/assignment_loss‚1.5177196060825964 1loss/SRqqt1/assignment_loss‚0.429647761343261 trainer/global_step‚15649 _runtime‚14166.240006026Š ¤„#IÀ1‚(:ô ©ÂöÒ„åèË" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚118956.61767578125 disk.dm-1.in‚0& proc.memory.availableMB‚ 485.984375# proc.memory.rssMB‚ 3024.74609375 cpu‚25.682733171413762 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 proc.cpu.threads‚17 network.sent‚ 2.5136807e+07) disk./.usagePercent‚76.04431092709987& disk.nvme0n1.out‚92881.58349609375& disk.nvme1c1n1.out‚42508.544921875" disk.dm-1.out‚15032.3603515625( disk.nvme0c0n1.out‚92881.58349609375) proc.memory.percent‚18.46158504486084 network.recv‚ 3.666273e+06% disk./.usageGB‚1167.4702835083008$ disk.nvme1n1.out‚42508.544921875 disk.md0.out‚ 671.3515625 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚97.03378677368164îmžÁ1‚(:“ ©ÂöÒÂã­Ì gpu.0.gpu‚62.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚278.751' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.91699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0δ-Â1‚(:ö ¸ÂöÒ–··Ë disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0" disk.dm-1.out‚15032.4365234375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚92883.87841796875& disk.nvme0n1.out‚92883.87841796875" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0" disk.dm-0.out‚120130.845703125 disk.rbd0.out‚0$ memory_percent‚97.03378677368164& proc.memory.availableMB‚ 485.984375 network.sent‚ 2.5185367e+07% disk./.usageGB‚1170.1882553100586 disk.nvme0n1.in‚ 309.37109375& disk.nvme1n1.out‚43684.01611328125) proc.memory.percent‚18.46158504486084 cpu‚25.683016833227455 proc.cpu.threads‚17 network.recv‚ 3.67066e+06) disk./.usagePercent‚76.22134866048256 disk.md0.out‚ 671.62890625# proc.memory.rssMB‚ 3024.74609375" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚43684.01611328125 disk.nvme1n1.in‚ 229.87109375ÆŽÙæžÃ1‚(:“ ¸ÂöÒàò†Ì gpu.0.gpu‚69.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚271.234' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.41133333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*+~OÄ1‚(:ö ÇÂöÒêÓœË network.sent‚ 2.5191323e+07& disk.nvme1n1.out‚44423.13427734375 disk.dm-0.in‚ 539.2421875 cpu‚25.683422294655756 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚92885.12158203125# proc.memory.rssMB‚ 3024.74609375 network.recv‚ 3.673986e+06) disk./.usagePercent‚76.22421803837042 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚92885.12158203125$ memory_percent‚97.03378677368164$ disk./.usageGB‚1170.232307434082 disk.md0.out‚ 672.16796875" disk.dm-0.out‚120872.865234375" disk.dm-1.out‚15032.5029296875" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚44423.13427734375& proc.memory.availableMB‚ 485.984375) proc.memory.percent‚18.46158504486084Y/·ŸžÅ1‚(:“ ÇÂöÒ¾¤ôË gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚271.277' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.42566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+=}4Æ1/ _wandbruntime‚14208  _runtime‚14208f6uÉÇ1à $ _timestamp‚1.784521035148139e+09 _runtime‚14208.723757768 _step‚1291 & lr-AdamW/pg1‚7.016365836326234e-05 & lr-AdamW/pg2‚7.016365836326234e-05 trainer/global_step‚15699–ìkÎÈ1È & lr-AdamW/pg1‚7.016365836326234e-05 & lr-AdamW/pg2‚7.016365836326234e-05 trainer/global_step‚15699 $ _timestamp‚1.784521035148139e+09 _runtime‚14208.723757768 _step‚1291‹ éxT4É1/ _wandbruntime‚14208  _runtime‚14208¯lË (Ê1…P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127143517 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127143515 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2389626026765712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18214659088854 _runtime‚14208.724650259 _step‚1292 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2389626009726352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155366441 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282509958 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2403696686895616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2407840328253932 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27389937 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2407840685854566 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18214318527580 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37470672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18242174682624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18344363807744 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 $ _timestamp‚1.784521035149146e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155356298 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2421938503114752 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚521939056674304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282499813 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18241833258496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155366441 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127143515 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27389940 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚521939038062080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18344254344192 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2403696669856256 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18241833258496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2421938861578240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2421938861578240 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2403696686895616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64860314 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2403696669856256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18242174682624 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344/½uÎ_Ë1ŠP _step‚1292 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2389626009726352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155366441 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282509958 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2403696686895616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2407840328253932 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27389937 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2407840685854566 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18214318527580 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37470672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18242174682624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18344363807744 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 $ _timestamp‚1.784521035149146e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155356298 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 NFDeviceStatsMonitor.on_train_”cg±batch_start/inactive_split.small_pool.peak‚511 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2421938503114752 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚521939056674304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282499813 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18241833258496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155366441 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127143515 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27389940 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚521939038062080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18344254344192 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2403696669856256 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18241833258496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2421938861578240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2421938861578240 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2403696686895616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64860314 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2403696669856256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18242174682624 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127143517 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127143515 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2389626026765712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18214659088854 _runtime‚14208.724650259Œ ¼ú®=4Ì1/ _wandbruntime‚14209  _runtime‚14209{r°{'Í1N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282527421 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚1293 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127151308 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2403844961397760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127151310 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92679168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18345380022784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚14209.339191637 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18242943357440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37472711 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64864659 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18215767477218 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2403844961397760 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2422087887715840 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18215426915832 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27391671 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2422088246186496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2403844944358400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27391674 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37472985 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2389773457433184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18243284788736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155376111 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282517262 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127151308 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚521972236944384 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155365954 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2403844944358400 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚540317598354944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155376111 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18242943357440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111291392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18345472701952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚521972218332160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2407988867309656 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2389773440393824 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64864382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚540317709646336 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚15699 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282527421 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845210357634127e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282517262 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127151310 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2422088246186496žÉ™'Î1”N N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18215767477218 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2403844961397760 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2422087887715840 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18215426915832 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27391671 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2422088246186496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2403844944358400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27391674 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37472985 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2389773457433184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18243284788736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155376111 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282517262 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127151308 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚521972236944384 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155365954 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2403844944358400 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚540317598354944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155376111 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18242943357440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111291392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18345472701952 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚521972218332160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2407988867309656 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2389773440393824 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64864382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚540317709646336 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚15699 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282527421 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845210357634127e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282517262 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127151310 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2422088246186496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 282527421 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚1293 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127151308 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2403844961397760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127151310 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92679168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18345380022784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚14209.339191637 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18242943357440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37472711 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64864659 H蜸4Ï1/ _wandbruntime‚14209  _runtime‚14209Ô“YÐ1‰ 0loss/SRqqt1/detection_loss‚0.610095594416753 1loss/SRqqt2/detection_loss‚0.6349170545067545 2loss/FBt1/assignment_loss‚0.006262585871728148 0loss/FRt1/detection_loss‚0.48797422358172376 /loss/FBt2/detection_loss‚0.3973994188513395 _runtime‚14209.339718417 /loss/FBt1/detection_loss‚0.5427139385543253 0loss/FRt2/assignment_loss‚0.6143727872556853 2loss/SRqqt2/assignment_loss‚0.5038029231562055 trainer/global_step‚15699 _step‚1294 epoch‚39 $ _timestamp‚1.784521035764411e+09 %loss/total_loss‚7.064523867806194 0loss/FRt1/assignment_loss‚1.4485815111694373 3loss/SRqqt1/assignment_loss‚0.36009262107105533 4loss/FBt2/assignment_loss‚0.00013348883387284227 /loss/FRt2/detection_loss‚1.4581777205373143 ݉Ñ1Ž _step‚1294 epoch‚39 $ _timestamp‚1.784521035764411e+09 %loss/total_loss‚7.064523867806194 0loss/FRt1/assignment_loss‚1.4485815111694373 3loss/SRqqt1/assignment_loss‚0.36009262107105533 4loss/FBt2/assignment_loss‚0.00013348883387284227 /loss/FRt2/detection_loss‚1.4581777205373143 0loss/SRqqt1/detection_loss‚0.610095594416753 1loss/SRqqt2/detection_loss‚0.6349170545067545 2loss/FBt1/assignment_loss‚0.006262585871728148 0loss/FRt1/detection_loss‚0.48797422358172376 /loss/FBt2/detection_loss‚0.3973994188513395 _runtime‚14209.339718417 /loss/FBt1/detection_loss‚0.5427139385543253 0loss/FRt2/assignment_loss‚0.6143727872556853 2loss/SRqqt2/assignment_loss‚0.5038029231562055 trainer/global_step‚15699Ž €¥³wqÒ1‚(:æ ÖÂöÒãä‹Ì$ memory_percent‚97.03400135040283# proc.memory.rssMB‚ 3024.74609375 network.recv‚ 3.67745e+06" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚672.5" disk.dm-1.out‚15032.6533203125 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚76.11008771291638# disk.nvme1n1.out‚44423.98828125# disk.dm-0.out‚120876.2314453125 disk.rbd0.in‚0( proc.memory.availableMB‚ 485.94921875) proc.memory.percent‚18.46158504486084 cpu‚25.683840368197135% disk./.usageGB‚1168.4801216125488 disk.dm-0.in‚ 539.2421875" disk.nvme0c0n1.in‚ 309.37109375# disk.nvme0n1.out‚92888.51953125 network.sent‚ 2.5209004e+07% disk.nvme1c1n1.out‚44423.98828125 disk.dm-1.in‚0% disk.nvme0c0n1.out‚92888.51953125 disk.nvme0n1.in‚ 309.37109375Áû—»žÓ1‚(:“ ÖÂöÒùèÓÌ gpu.0.gpu‚95.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.824' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.27466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ט…Ô1‚(:ú åÂöÒ¥·ØË" disk.nvme1c1n1.in‚ 229.87109375 cpu‚25.684297451161626 network.sent‚ 2.5259739e+07# disk.dm-0.out‚122777.0712890625& disk.nvme0n1.out‚94555.73779296875( disk.nvme1c1n1.out‚44662.01513671875 disk.nvme1n1.in‚ 229.87109375$ memory_percent‚97.03400135040283( proc.memory.availableMB‚ 485.94921875) proc.memory.percent‚18.46158504486084 network.recv‚ 3.681725e+06% disk./.usageGB‚1169.4411430358887 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 3024.74609375) disk./.usagePercent‚76.17268477680435 disk.dm-0.in‚ 539.2421875" disk.dm-1.out‚15035.6689453125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚94555.73779296875 disk.nvme0n1.in‚ 309.37109375& disk.nvme1n1.out‚44662.01513671875 proc.cpu.threads‚17 disk.md0.out‚ 673.15234375" disk.nvme0c0n1.in‚ 309.37109375%ÉqÕžÕ1‚(:“ åÂöÒ§ûÌ gpu.0.gpu‚52.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚223.036' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚74.34533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼!2¡Ö1‚(:ö ôÂöÒÖ›´Ë network.sent‚ 2.5264583e+07 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚122790.43505859375# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 cpu‚25.684700578641557 network.recv‚ 3.68496e+06( disk./.usagePercent‚75.9959775138573" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0c0n1.out‚94567.8623046875% disk.nvme0n1.out‚94567.8623046875% disk./.usageGB‚1166.7282447814941( disk.nvme1c1n1.out‚44664.30712890625& disk.nvme1n1.out‚44664.30712890625 disk.dm-1.in‚0" disk.dm-1.out‚15035.7626953125$ memory_percent‚97.03400135040283( proc.memory.availableMB‚ 485.94921875 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 673.6015625 proc.cpu.threads‚17W#°“×1‚(:ˆ ôÂöÒŒãÿË gpu.0.gpu‚97.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚122.127' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.709 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hƒI4Ø1/ _wandbruntime‚14252  _runtime‚14252‹o¨ÇÙ1Á trainer/global_step‚15749 $ _timestamp‚1.784521078495177e+09 _runtime‚14252.070794108 _step‚1295 % lr-AdamW/pg1‚6.86050810340554e-05 % lr-AdamW/pg2‚6.86050810340554e-05aqŽ¢ÌÚ1Æ _runtime‚14252.070794108 _step‚1295 % lr-AdamW/pg1‚6.86050810340554e-05 % lr-AdamW/pg2‚6.86050810340554e-05 trainer/global_step‚15749 $ _timestamp‚1.784521078495177e+09 åz±Ì4Û1/ _wandbruntime‚14252  _runtime‚14252„ûE·ÖÜ1…P TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_:^ÒM5'start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18297402360320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18297402360320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2411110412002816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18297743784448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2411110394963456 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚15749 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2429408155787264 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155840950 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18400111911424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚523598051566080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚14252.07165729 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155851093 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127533165 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155840950 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2429408155787264 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2429407797323776 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18297743784448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127533167 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2415267344840940 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127533165 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚541998289456128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2429407797323776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 283384260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27476637 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127533167 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚541998163477504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845210784961355e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155851093 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18269801740988@Ì(Ý1ŠP UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚15749 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2429408155787264 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155840950 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18400111911424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚523598051566080 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚14252.07165729 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155851093 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127533165 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155840950 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2429408155787264 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2429407797323776 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18297743784448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127533167 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2415267344840940 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127533165 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚541998289456128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2429407797323776 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 283384260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27476637 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127533167 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚541998163477504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845210784961355e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155851093 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18269801740988 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18297402360320 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18297402360320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2411110412002816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18297743784448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2411110394963456 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 283391564 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65065484 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚14252.683720473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18298512459264 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283401723 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65065764 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚542032466776576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127540960 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2429557181924864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127540958 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚523631250448384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27478374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27478371 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2397144973767424 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18298512459264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18270910129240 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37587113 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0Kû²9P à1“N VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 283391564 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65065484 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚14252.683720473 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18298512459264 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283401723 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65065764 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚542032466776576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127540960 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2429557181924864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127540958 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚523631250448384 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27478374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27478371 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2397144973767424 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18298512459264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18270910129240 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37587113 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18271250690626 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2429557540395520 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155850606 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2411258669465600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452‘ RãZ4á1/ _wandbruntime‚14252  _runtime‚14252·ŽYâ1Š 4loss/FBt1/assignment_loss‚0.00011774465847806613 /loss/FRt1/detection_loss‚0.4740180266631866 /loss/FRt2/detection_loss‚1.4516135068235405 1loss/SRqqt2/detection_loss‚0.6164185785955065 trainer/global_step‚15749 _runtime‚14252.684641574 1loss/SRqqt2/assignment_loss‚0.450921133791743 /loss/FBt2/detection_loss‚0.5193462969604536 0loss/FRt1/assignment_loss‚1.5360315960187179 0loss/FRt2/assignment_loss‚0.6183869067034976 1loss/SRqqt1/detection_loss‚0.5927251668032402 _step‚1298 3loss/FBt2/assignment_loss‚0.0002775215083475035 0loss/FBt1/detection_loss‚0.36784798841387645 %loss/total_loss‚7.088172635164219 % _timestamp‚1.7845210791093926e+09 epoch‚39 2loss/SRqqt1/assignment_loss‚0.4604681682236314 zŠeã1 epoch‚39 2loss/SRqqt1/assignment_loss‚0.4604681682236314 4loss/FBt1/assignment_loss‚0.00011774465847806613 /loss/FRt1/detection_loss‚0.4740180266631866 /loss/FRt2/detection_loss‚1.4516135068235405 1loss/SRqqt2/detection_loss‚0.6164185785955065 trainer/global_step‚15749 _runtime‚14252.684641574 1loss/SRqqt2/assignment_loss‚0.450921133791743 /loss/FBt2/detection_loss‚0.5193462969604536 0loss/FRt1/assignment_loss‚1.5360315960187179 0loss/FRt2/assignment_loss‚0.6183869067034976 1loss/SRqqt1/detection_loss‚0.5927251668032402 _step‚1298 3loss/FBt2/assignment_loss‚0.0002775215083475035 0loss/FBt1/detection_loss‚0.36784798841387645 %loss/total_loss‚7.088172635164219 % _timestamp‚1.7845210791093926e+09’ DC€¨vä1‚(:ë ƒÃöÒìÆÑË disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0% disk./.usageGB‚1166.7584114074707" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme1n1.out‚44664.93310546875 disk.md0.out‚ 674.0078125 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚122808.38427734375 disk.dm-1.in‚0# memory_percent‚97.0343828201294 network.sent‚ 2.5280444e+07 network.recv‚ 3.688286e+06 disk.nvme0n1.out‚ 94586.1875" disk.nvme1c1n1.in‚ 229.87109375 disk.rbd0.in‚0( proc.memory.availableMB‚ 485.88671875# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084! disk.nvme0c0n1.out‚ 94586.1875( disk.nvme1c1n1.out‚44664.93310546875" disk.dm-1.out‚15035.8955078125 disk.rbd0.out‚0 cpu‚25.685100880494186 proc.cpu.threads‚17) disk./.usagePercent‚75.99794244634234¹;Û¢žå1‚(:“ ƒÃöÒ¶‰Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.099' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.36633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î+Ù»~æ1‚(:ó ’ÃöÒÏàµË disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 486.08203125 network.recv‚ 3.69244e+06% disk./.usageGB‚1166.8731575012207" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚44667.17138671875 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0# disk.dm-1.out‚15036.21142578125" disk.nvme0c0n1.in‚ 309.37109375# disk.nvme0n1.out‚94639.89453125( disk.nvme1c1n1.out‚44667.17138671875# disk.dm-0.out‚122863.5517578125% disk.nvme0c0n1.out‚94639.89453125$ memory_percent‚97.03319072723389 disk.md0.out‚ 674.21484375# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 cpu‚25.68506487497667 proc.cpu.threads‚17 network.sent‚ 2.5332018e+07) disk./.usagePercent‚76.00541654461624¢ôüžç1‚(:“ ’ÃöÒ•‰ÿË gpu.0.gpu‚70.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.968' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.98933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã0–!4è1/ _wandbruntime‚14294  _runtime‚142942Ž<Éé1à $ _timestamp‚1.784521121167455e+09 _runtime‚14294.743049951 _step‚1299 & lr-AdamW/pg1‚6.706183108234277e-05 & lr-AdamW/pg2‚6.706183108234277e-05 trainer/global_step‚15799Ùä¼Îê1È _runtime‚14294.743049951 _step‚1299 & lr-AdamW/pg1‚6.706183108234277e-05 & lr-AdamW/pg2‚6.706183108234277e-05 trainer/global_step‚15799 $ _timestamp‚1.784521121167455e+09“ "¤¤4ë1/ _wandbruntime‚14294  _runtime‚14294l3(œ (ì1…P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18352971462144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65263308 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27563340 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65263016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2418524120070656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚525257065070080 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2418524120070656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2422694361427948 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18325284954396 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127922815 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156335745 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156325602 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚543713055608320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚15799 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127922815 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2436877449996288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37699968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚543713183684096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2436877449996288 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚14294.746796318 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18353312886272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _step‚1300 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2436877091532800 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2436877091532800 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18352971462144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65263308 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27563340 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65263016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2418524120070656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚525257065070080 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2418524120070656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2422694361427948 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18325284954396 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127922815 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156335745 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156325602 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚543713055608320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚15799 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127922815 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2436877449996288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37699968 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚543713183684096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>Devil]P¨ÎceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0” Kz­Hƒî1‚(:ø ¡ÃöÒšŽÝË network.sent‚ 2.5335041e+07% disk./.usageGB‚1166.8728065490723' disk.nvme1c1n1.out‚44667.6494140625 disk.nvme1n1.in‚ 229.87109375# disk.dm-1.out‚15036.24267578125 disk.rbd0.in‚0$ memory_percent‚97.03638553619385) disk./.usagePercent‚76.00539368500715% disk.nvme1n1.out‚44667.6494140625 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875# disk.dm-0.out‚122969.3759765625 disk.rbd0.out‚0( proc.memory.availableMB‚ 485.55859375# proc.memory.rssMB‚ 3024.74609375 network.recv‚ 3.695916e+06 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚94746.07177734375) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17& disk.nvme0n1.out‚94746.07177734375 disk.md0.out‚ 674.5859375 cpu‚25.684852283096195t-Î1žï1‚(:“ ¡ÃöÒ⊒Ì gpu.0.gpu‚55.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚289.599' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.53299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Íý;4ð1/ _wandbruntime‚14295  _runtime‚14295dµ2#'ñ1N U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2437026834604544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18354422992384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156335258 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18354422992384 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚543747480382976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284276025 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156345415 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚1301 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156335258 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18457216430592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65267049 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18326393342648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2404516524180384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127930608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ë4ï”'ò1•N B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2418672411612160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18457216430592 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65267049 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18326393342648 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2404516524180384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127930608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2437026834604544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18354422992384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156335258 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18354422992384 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚543747480382976 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284276025 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156345415 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚1301 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156335258 F‚ø1‚(:÷ ¿ÃöÒÄ–ÅË disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚122974.97607421875( proc.memory.availableMB‚ 486.05078125 network.sent‚ 2.5440786e+07 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0n1.out‚94751.64501953125' disk.nvme1c1n1.out‚44669.2646484375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 675.19140625 cpu‚25.68441399391332) disk./.usagePercent‚76.03809013174131% disk.nvme1n1.out‚44669.2646484375 disk.md0.in‚0$ memory_percent‚97.03338146209717 proc.cpu.threads‚17 disk.rbd0.out‚0( disk.nvme0c0n1.out‚94751.64501953125" disk.nvme1c1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 network.recv‚ 3.704175e+06% disk./.usageGB‚1167.3747787475586! disk.dm-1.out‚15036.501953125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375?Fžù1‚(:“ ¿ÃöÒÂË‹Ì gpu.0.gpu‚92.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚272.203' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.73433333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï„4n4ú1/ _wandbruntime‚14337  _runtime‚14337W‘VÊû1Ä & lr-AdamW/pg1‚6.553400762818488e-05 & lr-AdamW/pg2‚6.553400762818488e-05 trainer/global_step‚15849 % _timestamp‚1.7845211639422064e+09 _runtime‚14337.518278464 _step‚13035 „Ïü1É & lr-AdamW/pg2‚6.553400762818488e-05 trainer/global_step‚15849 % _timestamp‚1.7845211639422064e+09 _runtime‚14337.518278464 _step‚1303 & lr-AdamW/pg1‚6.553400762818488e-05— õTu4ý1/ _wandbruntime‚14337  _runtime‚14337D¦þ1„P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156820397 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18511910435328 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18380768167804 _runtime‚14337.52178934 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2444346385741824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚15849 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37814582 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚545427879545856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2444346744205312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128312465 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37814876 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2411740609847152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2425937862217216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18408540563968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128312467 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156810254 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚545428007621632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2425937862217216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMx‡donitor.on_train_batch_start/active.all.freed‚ 285122719 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128312465 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 285132864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2411740626886512 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚526916078574080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _step‚1304 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18408881988096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18408881988096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156810254 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2430121735615590 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2444346744205312 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2444346385741824 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128312467 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27650040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2430121378014956 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18511800971776 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156820397 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511;+ò;(ÿ1‰P XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2425937862217216 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18408540563968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128312467 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156810254 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚545428007621632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2425937862217216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 285122719 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128312465 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 285132864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2411740626886512 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚526916078574080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _step‚1304 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18408881988096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18408881988096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156810254 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚297 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚294 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2430121735615590 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2444346744205312 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2444346385741824 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128312467 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27650040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2430121378014956 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18511800971776 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156820397 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156820397 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18511910435328 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18380768167804 _runtime‚14337.52178934 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2444346385741824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚15849 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37814582 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚545427879545856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2444346744205312 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128312465 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37814876 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2411740609847152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1˜ y]>$4€2/ _wandbruntime‚14338  _runtime‚14338±–.'2N N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18382217117442 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2426086119680000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 trainer/global_step‚15849 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2426086119680000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18409650662912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚526949277456384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845211645566654e+09 _runtime‚14338.132400984 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 285150327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 285150327 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156830067 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚545462170446336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18409992094208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 285140168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156819910 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2444496128813568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27651771 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18512911602176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2444496128813568 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128320260 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65468627 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2430269917070680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37817148 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37816856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2430270274671426 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27651774 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128320258 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156819910 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128320260 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18409992094208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚545462302709248 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2411888040514624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2411888057553984 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18381876556056 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128320258 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 285140168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚526949258844160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18513025252864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156830067 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2444495770342912 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65468922 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 285150327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 285150327 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156830067 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚545462170446336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18409992094208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 285140168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156819910 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2444496128813568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27651771 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18512911602176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2444496128813568 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128320260 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65468627 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2430269917070680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37817148 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37816856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2430270274671426 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27651774 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128320258 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156819910 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128320260 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18409992094208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚545462302709248 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2411888040514624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2411888057553984 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18381876556056 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128320258 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 285140168 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚526949258844160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18513025252864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156830067 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2444495770342912 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65468922 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 35847 œ°Õû0656 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18382217117442 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2426086119680000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 trainer/global_step‚15849 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2426086119680000 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18409650662912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚526949277456384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799™ O ¡4ƒ2/ _wandbruntime‚14338  _runtime‚14338œH d „2„ /loss/FRt1/assignment_loss‚1.585960322416923 2loss/FBt1/assignment_loss‚0.028334617150668786 1loss/FBt2/assignment_loss‚0.01008358176455033 .loss/FRt2/detection_loss‚1.373441579093507 1loss/SRqqt1/detection_loss‚0.5807252286351285 /loss/FBt2/detection_loss‚0.7270649822435059 0loss/FRt2/assignment_loss‚0.5917026108737355 2loss/SRqqt1/assignment_loss‚0.3661882789677141 0loss/FRt1/detection_loss‚0.45094650856146895 1loss/SRqqt2/detection_loss‚0.5927649539897146 _step‚1306 trainer/global_step‚15849 $ _timestamp‚1.784521164559842e+09 _runtime‚14338.13576925 2loss/SRqqt2/assignment_loss‚0.4196204132680253 /loss/FBt1/detection_loss‚0.4781037518297576 &loss/total_loss‚7.2049368287946995 epoch‚39B®…2‰ 0loss/FRt2/assignment_loss‚0.5917026108737355 2loss/SRqqt1/assignment_loss‚0.3661882789677141 0loss/FRt1/detection_loss‚0.45094650856146895 1loss/SRqqt2/detection_loss‚0.5927649539897146 _step‚1306 trainer/global_step‚15849 $ _timestamp‚1.784521164559842e+09 _runtime‚14338.13576925 2loss/SRqqt2/assignment_loss‚0.4196204132680253 /loss/FBt1/detection_loss‚0.4781037518297576 &loss/total_loss‚7.2049368287946995 epoch‚39 /loss/FRt1/assignment_loss‚1.585960322416923 2loss/FBt1/assignment_loss‚0.028334617150668786 1loss/FBt2/assignment_loss‚0.01008358176455033 .loss/FRt2/detection_loss‚1.373441579093507 1loss/SRqqt1/detection_loss‚0.5807252286351285 /loss/FBt2/detection_loss‚0.7270649822435059š ½§6~†2‚(:ó ÎÃöÒ„¡ÄË disk.dm-0.in‚ 539.2421875" disk.dm-0.out‚124891.681640625" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚96668.57275390625 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚96668.57275390625& disk.nvme1n1.out‚44670.00537109375 disk.md0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚15036.6064453125 disk.rbd0.out‚0$ memory_percent‚97.03378677368164) proc.memory.percent‚18.46158504486084 network.sent‚ 2.544375e+07) disk./.usagePercent‚76.18525681637762 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 675.59375 disk.rbd0.in‚0& proc.memory.availableMB‚ 485.984375 proc.cpu.threads‚17" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚44670.00537109375# proc.memory.rssMB‚ 3024.74609375 cpu‚25.684325819009775 network.recv‚ 3.707698e+06% disk./.usageGB‚1169.6341552734375ÛŸ ž‡2‚(:“ ÎÃöÒ¤¼Ì gpu.0.gpu‚90.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚274.913' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.63766666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03ˆà~ˆ2‚(:ó ÝÃöÒÆÔ¸Ë) disk./.usagePercent‚76.18526352517594" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚44670.86181640625 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚44670.86181640625& disk.nvme0n1.out‚96669.71044921875) proc.memory.percent‚18.46158504486084 cpu‚25.68455126459523 network.sent‚ 2.5508075e+07 network.recv‚ 3.711533e+06" disk.dm-0.out‚124892.263671875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚96669.71044921875 disk.nvme0n1.in‚ 309.37109375 proc.cpu.threads‚17% disk./.usageGB‚1169.6342582702637" disk.dm-1.out‚15036.7666015625 disk.md0.in‚0 disk.md0.out‚676.1875 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚97.03593254089355' proc.memory.availableMB‚ 485.6328125# proc.memory.rssMB‚ 3024.74609375;‹$Þž‰2‚(:“ ÝÃöÒ‹ž„Ì gpu.0.gpu‚62.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.813' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.60433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·þ4 …Š2‚(:ú ìÃöÒ·Ÿ¾Ì network.recv‚ 3.714895e+06 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚97424.63330078125" disk.nvme1c1n1.in‚ 229.87109375( proc.memory.availableMB‚ 485.63671875 cpu‚25.684880117123836 network.sent‚ 2.5513019e+07) disk./.usagePercent‚76.18225798352964% disk./.usageGB‚1169.5881156921387 disk.rbd0.out‚0( disk.nvme0c0n1.out‚97424.63330078125& disk.nvme1n1.out‚44788.60693359375# proc.memory.rssMB‚ 3024.74609375 proc.cpu.threads‚17# disk.dm-1.out‚15036.83935546875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚44788.60693359375 disk.nvme1n1.in‚ 229.87109375$ memory_percent‚97.03590869903564) proc.memory.percent‚18.46158504486084 disk.md0.out‚ 676.421875 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚125764.35009765625Ùr+“‹2‚(:ˆ ìÃöÒäý¨Ì gpu.0.gpu‚65.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.005' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô¯I4Œ2/ _wandbruntime‚14382  _runtime‚14382@ ‘É2à $ _timestamp‚1.784521209077125e+09 _runtime‚14382.652767469 _step‚1307 & lr-AdamW/pg1‚6.402170880082735e-05 & lr-AdamW/pg2‚6.402170880082735e-05 trainer/global_step‚15899°<ÎŽ2È & lr-AdamW/pg2‚6.402170880082735e-05 trainer/global_step‚15899 $ _timestamp‚1.784521209077125e+09 _runtime‚14382.652767469 _step‚1307 & lr-AdamW/pg1‚6.402170880082735e-05› ½{š42/ _wandbruntime‚14382  _runtime‚14382Nƒ¹[ (2†P QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65666887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37929866 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2451816038414336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27736740 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18567615727104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18464451089920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286007166 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18567731482112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157294906 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2451815679950848 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128702115 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128702117 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2437548394601964 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157294906 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2419112160260112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128702115 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65666603 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344µ‰(‘2‹P XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65666603 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65666887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37929866 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2451816038414336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27736740 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18567615727104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18464451089920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286007166 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18567731482112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157294906 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2451815679950848 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128702115 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128702117 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2437548394601964 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157294906 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2419112160260112 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128702115 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0œ s½7Ô4’2/ _wandbruntime‚14383  _runtime‚14383±Õóš“2N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128709908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2433499844787200 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2437696933657688 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157314719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚528608290960384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18437359769464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65670593 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286024629 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286024629 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37932122 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚14383.279533215 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128709908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286014470 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2419259590927584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2433499844787200 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2433499861826560 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128709910 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128709910 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157314719 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18465219764736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27738474 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KB¾'”2•N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286024629 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286024629 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37932122 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚14383.279533215 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128709908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286014470 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2419259590927584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2433499844787200 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2433499861826560 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128709910 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128709910 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157314719 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18465219764736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27738474 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128709908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2433499844787200 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2437696933657688 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157314719 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚528608290960384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18437359769464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65670593 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ImÏ4•2/ _wandbruntime‚14383  _runtime‚14383Îêb –2† /loss/FRt1/assignment_loss‚1.501708850621299 4loss/FBt2/assignment_loss‚3.4804981815062506e-05 /loss/FRt1/detection_loss‚0.4569853294658661 0loss/SRqqt1/detection_loss‚0.656424280897989 _runtime‚14383.280085876 2loss/SRqqt1/assignment_loss‚0.3727704756980142 0loss/FBt2/detection_loss‚0.48946182019131984 $ _timestamp‚1.784521209704808e+09 0loss/FRt2/assignment_loss‚0.6745503175037878 2loss/SRqqt2/assignment_loss‚0.4582474159295778 .loss/FRt2/detection_loss‚1.398337347714308 1loss/SRqqt2/detection_loss‚0.7083668812077744 .loss/FBt1/detection_loss‚0.586499196428047 %loss/total_loss‚7.303755316358194 epoch‚39 trainer/global_step‚15899 3loss/FBt1/assignment_loss‚0.0003685957183949262 _step‚1310õŠyv—2‹ 0loss/FRt2/assignment_loss‚0.6745503175037878 2loss/SRqqt2/assignment_loss‚0.4582474159295778 .loss/FRt2/detection_loss‚1.398337347714308 1loss/SRqqt2/detection_loss‚0.7083668812077744 .loss/FBt1/detection_loss‚0.586499196428047 %loss/total_loss‚7.303755316358194 epoch‚39 trainer/global_step‚15899 3loss/FBt1/assignment_loss‚0.0003685957183949262 _step‚1310 /loss/FRt1/assignment_loss‚1.501708850621299 4loss/FBt2/assignment_loss‚3.4804981815062506e-05 /loss/FRt1/detection_loss‚0.4569853294658661 0loss/SRqqt1/detection_loss‚0.656424280897989 _runtime‚14383.280085876 2loss/SRqqt1/assignment_loss‚0.3727704756980142 0loss/FBt2/detection_loss‚0.48946182019131984 $ _timestamp‚1.784521209704808e+09ž þèTü˜2‚(:ô ûÃöÒ’§ÕË disk.dm-0.in‚ 539.2421875 cpu‚25.685259648119423 network.recv‚ 3.717968e+06) disk./.usagePercent‚76.18236457888068% disk./.usageGB‚1169.5897521972656% disk.nvme0n1.out‚97426.1845703125" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚44789.001953125$ disk.dm-0.out‚125765.68212890625 network.sent‚ 2.5528826e+07 disk.dm-1.in‚0# disk.dm-1.out‚15036.89013671875' disk.nvme0c0n1.out‚97426.1845703125 disk.md0.out‚ 676.6796875$ memory_percent‚97.03612327575684# proc.memory.rssMB‚ 3024.74609375) proc.memory.percent‚18.46158504486084 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.md0.in‚0' proc.memory.availableMB‚ 485.6015625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 309.37109375& disk.nvme1c1n1.out‚44789.001953125‘ŸŸ­ž™2‚(:“ ûÃöÒùúŸÌ gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚276.924' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.30799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝgZš2‚(:ö ŠÄöÒÕ›ÏË$ memory_percent‚97.03612327575684# proc.memory.rssMB‚ 3024.74609375 network.sent‚ 2.5580307e+07 network.recv‚ 3.722073e+06% disk.nvme1n1.out‚44789.8818359375 disk.md0.out‚ 677.13671875 disk.dm-0.in‚ 539.2421875' proc.memory.availableMB‚ 485.6015625 cpu‚25.685621981623434 proc.cpu.threads‚17 disk.nvme1n1.in‚ 229.87109375! disk.dm-0.out‚125776.20703125( disk.nvme0c0n1.out‚97436.94189453125 disk.nvme0n1.in‚ 309.37109375) proc.memory.percent‚18.46158504486084) disk./.usagePercent‚76.18351153491899% disk./.usageGB‚1169.6073608398438" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚44789.8818359375 disk.md0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.dm-1.in‚0# disk.dm-1.out‚15037.02978515625 disk.rbd0.in‚0& disk.nvme0n1.out‚97436.94189453125|õc›2‚(:’ ŠÄöÒû«Ì gpu.0.gpu‚29.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.41' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.13666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§è ntœ2‚(:é ™ÄöÒ•¦»Ë& disk.nvme0n1.out‚97450.27880859375" disk.nvme1c1n1.out‚ 44790.71875# disk.dm-1.out‚15037.08056640625( disk.nvme0c0n1.out‚97450.27880859375) proc.memory.percent‚18.46158504486084 proc.cpu.threads‚17 network.sent‚ 2.5585951e+07 disk.dm-0.out‚ 125789.40625 disk.rbd0.out‚0 cpu‚25.68600131137728% disk./.usageGB‚1169.7663803100586 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375" disk.nvme0c0n1.in‚ 309.37109375$ memory_percent‚97.03614711761475( proc.memory.availableMB‚ 485.59765625# proc.memory.rssMB‚ 3024.74609375) disk./.usagePercent‚76.19386942257343 disk.nvme1n1.in‚ 229.87109375 disk.nvme1n1.out‚ 44790.71875 disk.md0.in‚0 disk.md0.out‚ 677.56640625 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0 disk.rbd0.in‚0 network.recv‚ 3.731243e+06üýî°‘2‚(:† ™ÄöÒøœÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.235' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\'Xz4ž2/ _wandbruntime‚14428  _runtime‚14428}IXÊŸ2Ä & lr-AdamW/pg1‚6.252503173239795e-05 & lr-AdamW/pg2‚6.252503173239795e-05 trainer/global_step‚15949 % _timestamp‚1.7845212546831894e+09 _runtime‚14428.258786853 _step‚1311`ÞyÏ 2É trainer/global_step‚15949 % _timestamp‚1.7845212546831894e+09 _runtime‚14428.258786853 _step‚1311 & lr-AdamW/pg1‚6.252503173239795e-05 & lr-AdamW/pg2‚6.252503173239795e-05Ÿ ¶v-¸4¡2/ _wandbruntime‚14428  _runtime‚14428›((/ (¢2†P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18519678767616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2426483693633712 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2459285332623360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18519678767616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚14428.259880285 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 157779558 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38044567 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2426483676594352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2440765312431616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚548857599053824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27823437 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286871323 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 % _timestamp‚1.7845212546842709e+09 _step‚1312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18623493471744 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18520020191744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129091765 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚530234124194304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65868293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2459284974159872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18520020191744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18491734594620 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚15949 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18623602935296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129091767 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157789701 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2440765312431616 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286881468 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2440765295392256 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129091767 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157789701 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38044853 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18492075155894 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚548857727129600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2459284974159872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2444975768789606 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2444975411188972 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2459284974159872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18520020191744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18491734594620 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_trG¯•ê½$ain_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚15949 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18623602935296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129091767 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157789701 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2440765312431616 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286881468 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2440765295392256 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129091767 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157789701 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38044853 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18492075155894 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚548857727129600 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2459284974159872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2444975768789606 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2444975411188972 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2440765312431616 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚548857599053824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27823437 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286871323 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 % _timestamp‚1.7845212546842709e+09 _step‚1312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18623493471744 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18520020191744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129091765 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚530234124194304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65868293 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18521130297856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18624716433920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2459434717231616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2440913586933760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18520788866560 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129099558 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157799371 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65872337 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2459434358760960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18520788866560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18521130297856 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157799371 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2440913569894400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65872048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 _step‚1313 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2440913569894400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 % _timestamp‚1.7845212553160574e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129099560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27825174 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27825171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548892020898304 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2426631107261824 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286898931 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286888772 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚548891897024000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38047163 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18521130297856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18624716433920 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2459434717231616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2440913586933760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18520788866560 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129099558 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157799371 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65872337 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2459434358760960 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18520788866560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18521130297856 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157799371 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2440913569894400 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65872048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 _step‚1313 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2440913569894400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 % _timestamp‚1.7845212553160574e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129099560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27825174 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27825171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548892020898304 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2426631107261824 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286898931 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286888772 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚548891897024000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38047163 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2452402785376614 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158264210 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18575247869440 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27910137 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2466754268368896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18575589293568 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18575589293568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158264210 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18679436964864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2448179037538816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38160172 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2452402427775980 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0IÔ(µ2‹P RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2466754268368896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18575589293568 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18575589293568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158264210 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18679436964864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2448179037538816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38160172 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2452402427775980 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2466754626832384 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2466754268368896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129481415 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚531893119086080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18547217808028 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129481415 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287755770 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66070022 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2448179020499456 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18575247869440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚15999 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158274353 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129481417 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2452402785376614 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158264210 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18575247869440 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27910137 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528¤ å±¾4¶2/ _wandbruntime‚14474  _runtime‚14474ç}rY$·2N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18548326196280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66073988 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27911874 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129489210 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18680550881280 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2452550966831704 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚15999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158273866 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2448327295001600 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18576699399680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38162117 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2434002640635424 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158273866 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287773233 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287763074 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2448327295001600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845213009017758e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2466904011440640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2466903652969984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18576699399680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129489210 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2448327312040960 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _step‚1317sTº£'¸2•N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158273866 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287773233 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287763074 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2448327295001600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845213009017758e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2466904011440640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2466903652969984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18576699399680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129489210 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2448327312040960 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _step‚1317 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18548326196280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66073988 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27911874 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129489210 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18680550881280 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2452550966831704 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚15999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158273866 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2448327295001600 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18576699399680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38162117 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2434002640635424 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184¥ ó¶ØB4¹2/ _wandbruntime‚14474  _runtime‚14474SNsº2‰ 3loss/SRqqt1/assignment_loss‚0.39757562581590195 2loss/SRqqt2/assignment_loss‚0.5200800202197404 1loss/SRqqt1/detection_loss‚0.6417755423692065 epoch‚39 _runtime‚14474.478802429 _step‚1318 .loss/FRt1/assignment_loss‚1.50937747119078 4loss/FBt1/assignment_loss‚0.00023085922066011394 3loss/FBt2/assignment_loss‚0.0019842375887364483 .loss/FRt1/detection_loss‚0.506627318562036 /loss/FBt1/detection_loss‚0.4292726622881088 %loss/total_loss‚7.579331107316114 0loss/FRt2/assignment_loss‚0.8452728004934632 /loss/FRt2/detection_loss‚1.5976935632485532 1loss/SRqqt2/detection_loss‚0.6476729367121672 0loss/FBt2/detection_loss‚0.48176806960675983 trainer/global_step‚15999 % _timestamp‚1.7845213009033763e+09AV4:»2Ž 3loss/SRqqt1/assignment_loss‚0.39757562581590195 2loss/SRqqt2/assignment_loss‚0.5200800202197404 1loss/SRqqt1/detection_loss‚0.6417755423692065 epoch‚39 _runtime‚14474.478802429 _step‚1318 .loss/FRt1/assignment_loss‚1.50937747119078 4loss/FBt1/assignment_loss‚0.00023085922066011394 3loss/FBt2/assignment_loss‚0.0019842375887364483 .loss/FRt1/detection_loss‚0.506627318562036 /loss/FBt1/detection_loss‚0.4292726622881088 %loss/total_loss‚7.579331107316114 0loss/FRt2/assignment_loss‚0.8452728004934632 /loss/FRt2/detection_loss‚1.5976935632485532 1loss/SRqqt2/detection_loss‚0.6476729367121672 0loss/FBt2/detection_loss‚0.48176806960675983 trainer/global_step‚15999 % _timestamp‚1.7845213009033763e+09¦ ̓¼2‚(:ø ÕÄöÒ°þÌ) proc.memory.percent‚17.27125644683838 cpu‚25.687471438457628 proc.cpu.threads‚17 network.sent‚ 2.5664473e+07) disk./.usagePercent‚76.19063379399222# disk.dm-0.out‚125953.1904296875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 678.64453125$ memory_percent‚96.56286239624023 network.recv‚ 3.746085e+06 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375 disk.md0.in‚0# proc.memory.rssMB‚ 2829.72265625 disk.dm-0.in‚ 539.2421875& disk.nvme0n1.out‚97613.21826171875" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚45329.21630859375& disk.nvme1n1.out‚45329.21630859375% disk./.usageGB‚1169.7167053222656" disk.dm-1.out‚15572.3291015625( disk.nvme0c0n1.out‚97613.21826171875& proc.memory.availableMB‚ 563.1406253Þ­öž½2‚(:“ ÕÄöÒ‘ùóÌ gpu.0.gpu‚95.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚289.411' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.47033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷+òv¾2‚(:ë äÄöÒ×ׯË disk.nvme0n1.out‚ 97616.09375) disk./.usagePercent‚76.19063429094025$ disk.dm-0.out‚125955.86865234375 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme0c0n1.out‚ 97616.09375' disk.nvme1c1n1.out‚45330.3447265625' proc.memory.availableMB‚ 443.0078125* proc.memory.percent‚18.002867698669434 disk.rbd0.out‚0 network.sent‚ 2.5727696e+07% disk./.usageGB‚1169.7167129516602 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15572.44775390625 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 network.recv‚ 3.750048e+06 disk.md0.in‚0$ memory_percent‚97.29609489440918# proc.memory.rssMB‚ 2949.58984375 cpu‚25.690623792095483 proc.cpu.threads‚17% disk.nvme1n1.out‚45330.3447265625 disk.md0.out‚ 679.21875 disk.rbd0.in‚0»¥.7ž¿2‚(:“ äÄöÒƪìË gpu.0.gpu‚71.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.389' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.12966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡2L4À2/ _wandbruntime‚14492  _runtime‚14492ì”ì y=Á2óz =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9962487816810608 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7030419707298279 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3366888761520386 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27265995740890503 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968310594558716 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.208934485912323 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21225053071975708 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5657673478126526 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6595205664634705 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07435201853513718 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5890985727310181 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6322417855262756 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2896985709667206 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6439619064331055 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7882653474807739 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7531596422195435 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08254092931747437 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6930769085884094 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6106838583946228 particle/accuracy_2_of_2‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.885054349899292 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30457061529159546 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967972636222839 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7065109014511108 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2926170825958252 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3479965627193451 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3419329822063446 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24957716464996338 +particle/sensitivity‚0.6402226686477661 % _timestamp‚1.7845213184642522e+09 _step‚1319 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6094468235969543 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.988936722278595 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.613432765007019 )jet/accuracy_1_of_2‚0.752558708190918 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10501790046691895 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9110103845596313 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6881831884384155 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6277791261672974 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23017294704914093 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2873951196670532 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6949935555458069 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04270750284194946 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882233738899231 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04868609458208084 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5924740433692932 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0542798712849617 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19372761249542236 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6343106627464294 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8490362763404846 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30024439096450806 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 )jet/accuracy_1_of_4‚0.962018609046936 +jet/accuracy_3_of_3‚0.07515798509120941 particle/accuracy_5_of_5‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.989250659942627 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5687670707702637 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4333035945892334 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6923725008964539 epoch‚39 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2622598707675934 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9907164573669434 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2926170825958252 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5654541850090027 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6608293056488037 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29482215642929077 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08263777196407318 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925374031067 6validation_average_jet_accuracy‚0.3670100271701813 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33646565675735474 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6672182083129883 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +particle/specificity‚0.9089258313179016 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6760410070419312 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7072641253471375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2835672199726105 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8773620128631592 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6821550130844116 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995364785194397 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5914207696914673 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8626417517662048 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5004037022590637 particle/accuracy_3_of_5‚1 5 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.85309898853302 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7588713765144348 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5867547392845154 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08627661317586899 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6355111002922058 particle/accuracy_2_of_5‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2665998041629791 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.29256322979927063 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5853575468063354 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4463428258895874 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30371958017349243 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0831252858042717 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9117426872253418 +jet/accuracy_4_of_4‚0.01670061983168125 trainer/global_step‚15999 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5059849619865417 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31192225217819214 /particle/accuracy_3_of_3‚0.9994120597839355 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _runtime‚14492.040393045 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26090872287750244 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2227938324213028 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9778733849525452 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19850556552410126 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07435201853513718 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5849135518074036 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30351635813713074 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5528851747512817 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5364583134651184 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7531596422195435 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2749466300010681 +jet/accuracy_3_of_4‚0.09184560924768448 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6125865578651428 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03752489760518074 /particle/accuracy_4_of_4‚0.9535804986953735 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211899042129517 *jet/accuracy_2_of_2‚0.1550668478012085 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8723439574241638 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.58400559425354 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2733801007270813 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.22666668891906738 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7320296168327332 *jet/accuracy_1_of_1‚0.3285674452781677 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6088250279426575 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8425925374031067 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6488924026489258 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7526041865348816 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5657673478126526 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08531566709280014 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6576536893844604 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.499062716960907 7"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.64952552318573 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2202526330947876 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8512176275253296 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9961062669754028 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 )loss/val_total_loss‚7.421151534493168 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1974131464958191 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6678994297981262 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5883333086967468 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9468596577644348 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685941100120544 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_3‚0.5430648326873779 )jet/accuracy_2_of_4‚0.728640615940094 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4614379107952118 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837QW±Ã›Â2øz *jet/accuracy_2_of_3‚0.5430648326873779 )jet/accuracy_2_of_4‚0.728640615940094 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4614379107952118 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9962487816810608 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7030419707298279 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3366888761520386 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27265995740890503 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968310594558716 6 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.208934485912323 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21225053071975708 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5657673478126526 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6595205664634705 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07435201853513718 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5890985727310181 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6322417855262756 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2896985709667206 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6439619064331055 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7882653474807739 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7531596422195435 =&Purity/*FRt0SRqq!ê9ã5t2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08254092931747437 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6930769085884094 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6106838583946228 particle/accuracy_2_of_2‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.885054349899292 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30457061529159546 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967972636222839 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7065109014511108 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2926170825958252 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3479965627193451 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3419329822063446 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24957716464996338 +particle/sensitivity‚0.6402226686477661 % _timestamp‚1.7845213184642522e+09 _step‚1319 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6094468235969543 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.988936722278595 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.613432765007019 )jet/accuracy_1_of_2‚0.752558708190918 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10501790046691895 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9110103845596313 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6881831884384155 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6277791261672974 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23017294704914093 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2873951196670532 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6949935555458069 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04270750284194946 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9882233738899231 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04868609458208084 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5924740433692932 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0542798712849617 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19372761249542236 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6343106627464294 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8490362763404846 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30024439096450806 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 )jet/accuracy_1_of_4‚0.962018609046936 +jet/accuracy_3_of_3‚0.07515798509120941 particle/accuracy_5_of_5‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.989250659942627 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5687670707702637 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4333035945892334 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6923725008964539 epoch‚39 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2622598707675934 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9907164573669434 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2926170825958252 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5654541850090027 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6608293056488037 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29482215642929077 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08263777196407318 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925374031067 6validation_average_jet_accuracy‚0.3670100271701813 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99916011095047 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33646565675735474 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6672182083129883 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +particle/specificity‚0.9089258313179016 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6760410070419312 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7072641253471375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2835672199726105 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8773620128631592 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6821550130844116 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995364785194397 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5914207696914673 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8626417517662048 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.5004037022590637 particle/accuracy_3_of_5‚1 5 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.85309898853302 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7588713765144348 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5867547392845154 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08627661317586899 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6355111002922058 particle/accuracy_2_of_5‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2665998041629791 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.29256322979927063 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5853575468063354 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4463428258895874 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30371958017349243 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_4_of_5‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0831252858042717 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9117426872253418 +jet/accuracy_4_of_4‚0.01670061983168125 trainer/global_step‚15999 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5059849619865417 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31192225217819214 /particle/accuracy_3_of_3‚0.9994120597839355 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 _runtime‚14492.040393045 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26090872287750244 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2227938324213028 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9778733849525452 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19850556552410126 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987265467643738 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07435201853513718 particle/accuracy_2_of_4‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5849135518074036 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30351635813713074 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5528851747512817 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5364583134651184 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7531596422195435 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 particle/accuracy_1_of_3‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2749466300010681 +jet/accuracy_3_of_4‚0.09184560924768448 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6125865578651428 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03752489760518074 /particle/accuracy_4_of_4‚0.9535804986953735 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211899042129517 *jet/accuracy_2_of_2‚0.1550668478012085 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8723439574241638 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.58400559425354 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2733801007270813 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.22666668891906738 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7320296168327332 *jet/accuracy_1_of_1‚0.3285674452781677 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6088250279426575 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8425925374031067 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6488924026489258 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7526041865348816 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5657673478126526 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08531566709280014 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6576536893844604 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.499062716960907 7"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.64952552318573 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2202526330947876 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8512176275253296 jet/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9961062669754028 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 )loss/val_total_loss‚7.421151534493168 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1974131464958191 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6678994297981262 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5883333086967468 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9468596577644348 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8685941100120544 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648§ 2ÙÿÆÃ2‚(:ö óÄöÒáÒœË% disk./.usageGB‚1169.7168045043945 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0# disk.dm-1.out‚15572.55322265625 proc.cpu.threads‚17 disk.md0.in‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚45331.021484375* proc.memory.percent‚18.014931678771973 disk.rbd0.out‚0' disk.nvme0c0n1.out‚97617.1884765625 disk.nvme0n1.in‚ 309.37109375$ memory_percent‚97.30994701385498( proc.memory.availableMB‚ 440.73828125 cpu‚25.69149484539802 network.recv‚ 3.752908e+06 disk.md0.out‚ 679.69140625$ disk.dm-0.out‚125956.53271484375 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375% disk.nvme0n1.out‚97617.1884765625& disk.nvme1c1n1.out‚45331.021484375# proc.memory.rssMB‚ 2951.56640625 network.sent‚ 2.5730618e+07) disk./.usagePercent‚76.19064025431653¿h7ÐÄ2‚(:’ óÄöÒéî”Ë gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚267.49' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.16333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëM€Å2‚(:õ ‚ÅöÒŠ§µË) disk./.usagePercent‚76.19069094301493 disk.nvme0n1.in‚ 309.37109375 disk.dm-1.in‚0( proc.memory.availableMB‚ 440.98828125* proc.memory.percent‚18.014931678771973 cpu‚25.691778082310634 network.recv‚ 3.757767e+06% disk./.usageGB‚1169.7175827026367" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 680.046875 disk.dm-0.out‚125957.9140625$ memory_percent‚97.30842113494873# proc.memory.rssMB‚ 2951.56640625 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚97618.75048828125' disk.nvme1c1n1.out‚45331.7998046875 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚45331.7998046875 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875 network.sent‚ 2.5788078e+07( disk.nvme0c0n1.out‚97618.75048828125# disk.dm-1.out‚15572.72119140625 proc.cpu.threads‚17a4`ržÆ2‚(:“ ‚ÅöÒóè¥Ì gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚269.309' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.76966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x0¡¼}Ç2‚(:ò ‘ÅöÒ›£¸Ë network.recv‚ 3.767729e+06( disk./.usagePercent‚76.1906842342166" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0! disk.dm-0.out‚125959.16796875( proc.memory.availableMB‚ 440.98828125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.out‚ 680.2890625 disk.dm-0.in‚ 539.2421875$ memory_percent‚97.30842113494873# proc.memory.rssMB‚ 2951.56640625* proc.memory.percent‚18.014931678771973 proc.cpu.threads‚17& disk.nvme1c1n1.out‚45332.263671875$ disk.nvme1n1.out‚45332.263671875# disk.dm-1.out‚15572.77978515625 cpu‚25.6920785425646 network.sent‚ 2.5798185e+07% disk./.usageGB‚1169.7174797058105( disk.nvme0c0n1.out‚97620.12841796875& disk.nvme0n1.out‚97620.12841796875 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0€[BUÈ2‚(:’ ‘ÅöÒÈöË gpu.0.gpu‚73.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.81' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.93666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬¶k4É2/ _wandbruntime‚14537  _runtime‚14537‰@‹ÉÊ2à trainer/global_step‚16049 $ _timestamp‚1.784521363485217e+09 _runtime‚14537.060838898 _step‚1320 & lr-AdamW/pg1‚5.957892637787792e-05 & lr-AdamW/pg2‚5.957892637787792e-05,ŽxÎË2È & lr-AdamW/pg2‚5.957892637787792e-05 trainer/global_step‚16049 $ _timestamp‚1.784521363485217e+09 _runtime‚14537.060838898 _step‚1320 & lr-AdamW/pg1‚5.957892637787792e-05¨ ñ‡X?4Ì2/ _wandbruntime‚14537  _runtime‚14537ÌÈ]k (Í2†P UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2457535106521600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2476186582752768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2461771004220109 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2476186224289280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2476186224289280 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2476186582752768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚552330723036672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚16049 _step‚1321 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158830356 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66313318 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18755993171968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18623004870333 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 288824229 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18755887902720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2457535089482240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 288834374 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129993875 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚533574711252480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18651134807040 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38312078 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158840499 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2443148016389136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129993875 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2443147999349776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2457535106521600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158840499 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38312362 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158830356 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66313031 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28000953 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚552330599155200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0F¢‹àaÎ2‹P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2461771004220109 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2476186224289280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2476186224289280 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2476186582752768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚552330723036672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚16049 _step‚1321 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158830356 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66313318 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18755993171968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18623004870333 ?9DeviceStatsMonitor.on_train_batch_start/num_eì°alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 288824229 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18755887902720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2457535089482240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 288834374 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129993875 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚533574711252480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18651134807040 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38312078 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158840499 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2443148016389136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129993875 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2443147999349776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2457535106521600 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158840499 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38312362 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158830356 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66313031 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28000953 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚552330599155200 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2457535106521600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2476186582752768© •jº4Ï2/ _wandbruntime‚14537  _runtime‚14537†¼]9'Ð2N FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130001668 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2443295430017248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚533607910134784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28002690 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130001666 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158850169 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 % _timestamp‚1.7845213641150827e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 288851837 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66317392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28002687 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18624453819971 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18624113258585 _runtime‚14537.690920202 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2476335967361024 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2476335967361024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 288841678 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158840012 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2457683381023744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158840012 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚16049 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2457683381023744 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130001666 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2443295447056608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130001668 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 288851837 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66317105 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18652586337280 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2457683363984384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _step‚1322 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2461919900876579 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130001668 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2443295430017248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚533607910134784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94776320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28002690 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130001666 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158850169 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 % _timestamp‚1.7845213641150827e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 288851837 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66317392 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28002687 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18624453819971 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18624113258585 _runtime‚14537.690920202 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2476335967361024 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2476335967361024 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 288841678 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158840012 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2457683381023744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158840012 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚16049 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113388544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2457683381023744 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130001666 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2443295447056608 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130001668 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 288851837 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66317105 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746ª ~&û4Ò2/ _wandbruntime‚14537  _runtime‚14537Y ß: Ó2‡ trainer/global_step‚16049 _runtime‚14537.692186704 epoch‚40 1loss/FBt1/assignment_loss‚0.01621233383927516 2loss/FBt2/assignment_loss‚0.003271718206219109 $ _timestamp‚1.784521364116731e+09 0loss/FRt2/assignment_loss‚0.6249593415043571 .loss/FRt1/detection_loss‚0.457416581966834 /loss/FRt2/detection_loss‚1.4467707537632022 1loss/SRqqt2/detection_loss‚0.5750458243224809 %loss/total_loss‚7.253879917242077 0loss/FRt1/assignment_loss‚1.5426006007171769 3loss/SRqqt1/assignment_loss‚0.39852650813366686 3loss/SRqqt2/assignment_loss‚0.45678855786781003 1loss/SRqqt1/detection_loss‚0.6144861572331541 0loss/FBt1/detection_loss‚0.37472364731606345 /loss/FBt2/detection_loss‚0.7430778923718369 _step‚1323—Ô2Œ trainer/global_step‚16049 _runtime‚14537.692186704 epoch‚40 1loss/FBt1/assignment_loss‚0.01621233383927516 2loss/FBt2/assignment_loss‚0.003271718206219109 $ _timestamp‚1.784521364116731e+09 0loss/FRt2/assignment_loss‚0.6249593415043571 .loss/FRt1/detection_loss‚0.457416581966834 /loss/FRt2/detection_loss‚1.4467707537632022 1loss/SRqqt2/detection_loss‚0.5750458243224809 %loss/total_loss‚7.253879917242077 0loss/FRt1/assignment_loss‚1.5426006007171769 3loss/SRqqt1/assignment_loss‚0.39852650813366686 3loss/SRqqt2/assignment_loss‚0.45678855786781003 1loss/SRqqt1/detection_loss‚0.6144861572331541 0loss/FBt1/detection_loss‚0.37472364731606345 /loss/FBt2/detection_loss‚0.7430778923718369 _step‚1323« .èºb~Õ2‚(:ó  ÅöÒ§ÅæË network.recv‚ 3.771184e+06$ disk.dm-0.out‚125961.22216796875 disk.dm-1.in‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375# proc.memory.rssMB‚ 2951.56640625* proc.memory.percent‚18.014931678771973 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚45332.86083984375 disk.md0.in‚0 disk.md0.out‚ 680.48828125$ memory_percent‚97.31011390686035" disk.nvme0n1.out‚ 97622.1484375) disk./.usagePercent‚76.19067454373015% disk./.usageGB‚1169.7173309326172 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 97622.1484375& disk.nvme1n1.out‚45332.86083984375 proc.cpu.threads‚17 network.sent‚ 2.5813232e+07" disk.dm-1.out‚15572.8876953125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375' proc.memory.availableMB‚ 440.7109375 cpu‚25.692462943700185Ô«œÖ2‚(:‘  ÅöÒëð¬Ì gpu.0.gpu‚2.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚239.905' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚79.96833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö3^€×2‚(:õ ¯ÅöÒÜŠÔË( disk./.usagePercent‚76.1906777738923" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚45333.40087890625 disk.md0.out‚ 680.8515625$ disk.dm-0.out‚125961.86669921875* proc.memory.percent‚18.014931678771973 proc.cpu.threads‚17 network.recv‚ 3.776128e+06" disk.dm-1.out‚15572.9384765625& disk.nvme1n1.out‚45333.40087890625 disk.dm-0.in‚ 539.2421875# proc.memory.rssMB‚ 2951.56640625 network.sent‚ 2.5865596e+07 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375# memory_percent‚97.3085880279541' proc.memory.availableMB‚ 440.9609375 cpu‚25.69274555667971% disk./.usageGB‚1169.7173805236816 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚97623.1103515625 disk.nvme0n1.in‚ 309.37109375% disk.nvme0n1.out‚97623.1103515625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0›ý*žØ2‚(:“ ¯ÅöÒñª–Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚278.403' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.80100000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìy|}ƒÙ2‚(:ø ¾ÅöÒ¥üµË disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚97632.23291015625$ memory_percent‚97.30861186981201( proc.memory.availableMB‚ 440.95703125 network.sent‚ 2.587044e+07 network.recv‚ 3.779791e+06( disk.nvme1c1n1.out‚45335.47900390625 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375# proc.memory.rssMB‚ 2951.56640625* proc.memory.percent‚18.014931678771973 cpu‚25.693113574272665 disk.nvme0n1.in‚ 309.37109375 proc.cpu.threads‚17" disk.dm-0.out‚125972.142578125) disk./.usagePercent‚76.17514665731291% disk./.usageGB‚1169.4789390563965& disk.nvme0n1.out‚97632.23291015625" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0! disk.dm-1.out‚15573.099609375& disk.nvme1n1.out‚45335.47900390625 disk.md0.in‚0 disk.md0.out‚ 681.20703125€šžÚ2‚(:“ ¾ÅöÒĦ…Ì gpu.0.gpu‚47.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚274.236' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.41199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w¯4Û2/ _wandbruntime‚14582  _runtime‚14582PþþGÊÜ2Ä & lr-AdamW/pg1‚5.812968731462915e-05 & lr-AdamW/pg2‚5.812968731462915e-05 trainer/global_step‚16099 % _timestamp‚1.7845214086999216e+09 _runtime‚14582.275491981 _step‚1324‹Q ÏÝ2É _runtime‚14582.275491981 _step‚1324 & lr-AdamW/pg1‚5.812968731462915e-05 & lr-AdamW/pg2‚5.812968731462915e-05 trainer/global_step‚16099 % _timestamp‚1.7845214086999216e+09¬ LF&û4Þ2/ _wandbruntime‚14582  _runtime‚14582ÌÃbYÂß2…P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130383523 ICDeviceStatsMonitor.on_tr?¢EI&ain_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159325151 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130383523 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2464948814589440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2464948831628800 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130383525 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚535233724756480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18811783629312 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2464948831628800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159315008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28087653 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2483655518498304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289708676 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2483655876961792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66515312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28087656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚535233743368704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2469198378407751 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66515589 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2483655518498304 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚554045508385792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2469198020807117 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚554045625975808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2483655518498304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289708676 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2483655876961792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66515312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28087656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚535233743368704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2469198378407751 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66515589 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2483655518498304 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚554045508385792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2469198020807117 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚554045625975808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JÄù'â2N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159334821 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚554079798139392 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535266905026560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 $ _timestamp‚1.784521409331034e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28089387 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2465097106130944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18813006763520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2483805261570048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18679596471993 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159324664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38430205 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚554079930402304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1326 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289715980 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130391318 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2450666963390848 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66519292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289715980 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159334821 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535266923638784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2465097089091584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289726139 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚14582.906834176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18707814007808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66519595 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2465097106130944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18707814007808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2465097089091584 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18708155439104 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2450666980430208 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130391318 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28089390 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130391316 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38429905 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289726139 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚16099 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130391316 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2483805261570048 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18812893112832 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2469346559862841Á±Ôã; ã2”N =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18679596471993 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159324664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38430205 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚554079930402304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1326 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289715980 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130391318 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2450666963390848 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E?Device$ ózßStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66519292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289715980 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159334821 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535266923638784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2465097089091584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289726139 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚14582.906834176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18707814007808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66519595 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2465097106130944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18707814007808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2465097089091584 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18708155439104 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2450666980430208 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130391318 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28089390 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130391316 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38429905 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289726139 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚16099 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130391316 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2483805261570048 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18812893112832 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2469346559862841 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159334821 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚554079798139392 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535266905026560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 $ _timestamp‚1.784521409331034e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28089387 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2465097106130944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18813006763520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2483805261570048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚300 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18867624939008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130773173 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2472362539696640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 290582978 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159809803 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290572833 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2491124812707328 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130773175 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚536892738260480 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2476625394994759 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2457891083136336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18867736499712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38542924 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18762273010688 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18762614434816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚555760363199488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2472362539696640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159799660 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274¥ïDë"ò2ŠP aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚536892738260480 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2476625394994759 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2457891083136336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18867736499712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38542924 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18762273010688 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18762614434816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚555760363199488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2472362539696640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159799660 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18867624939008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130773173 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2472362539696640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 290582978 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159809803 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290572833 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2491124812707328 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130773175± A–[4ó2/ _wandbruntime‚14628  _runtime‚146286¢Ï'ô2ŽN SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130780968 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2476773576449849 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130780966 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66721042 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2458038496764448 _runtime‚ 14628.1671193 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2491274197308416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130780968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38545244 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18868740808192 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18763383109632 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159809316 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2472510831238144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 % _timestamp‚1.7845214545912807e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290600441 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159809316 trainer/global_step‚16149 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1330 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159819473 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38544955 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2476773934050595 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290600441 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28176090 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18763724540928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28176087 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚536925937142784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚536925918530560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18735420246787 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2491274555779072 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2491274555779072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2458038513803808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚555794791601664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚555794659338752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290590282 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896ùcb±'õ2“N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚536925918530560 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18735420246787 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2491274555779072 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2491274555779072 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2458038513803808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚555794791601664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚555794659338752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290590282 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130780968 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2476773576449849 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130780966 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66721042 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2458038496764448 _runtime‚ 14628.1671193 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2491274197308416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130780968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38545244 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18868740808192 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18763383109632 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159809316 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2472510831238144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 % _timestamp‚1.7845214545912807e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290600441 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159809316 trainer/global_step‚16149 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1330 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159819473 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38544955 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2476773934050595 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290600441 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28176090 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18763724540928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28176087 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚536925937142784² §04ö2/ _wandbruntime‚14628  _runtime‚14628mÇ®(÷2Œ epoch‚40 3loss/FBt1/assignment_loss‚0.0005724050369132544 1loss/SRqqt1/detection_loss‚0.5692969765711146 1loss/SRqqt2/detection_loss‚0.6480625560651495 trainer/global_step‚16149 _runtime‚14628.168900533 0loss/FRt1/assignment_loss‚1.4117601290983206 4loss/FBt2/assignment_loss‚0.00014727541260878174 /loss/FRt2/detection_loss‚1.4204752983736382 % _timestamp‚1.7845214545933552e+09 _step‚1331 2loss/SRqqt1/assignment_loss‚0.4495325065376561 2loss/SRqqt2/assignment_loss‚0.4787031359352613 0loss/FRt1/detection_loss‚0.43232258960649833 /loss/FBt2/detection_loss‚0.5574419199885927 &loss/total_loss‚6.9416202237930325 0loss/FRt2/assignment_loss‚0.5948470283973135 /loss/FBt1/detection_loss‚0.3784584027699659hóø2‘ epoch‚40 3loss/FBt1/assignment_loss‚0.0005724050369132544 1loss/SRqqt1/detection_loss‚0.5692969765711146 1loss/SRqqt2/detection_loss‚0.6480625560651495 trainer/global_step‚16149 _runtime‚14628.168900533 0loss/FRt1/assignment_loss‚1.4117601290983206 4loss/FBt2/assignment_loss‚0.00014727541260878174 /loss/FRt2/detection_loss‚1.4204752983736382 % _timestamp‚1.7845214545933552e+09 _step‚1331 2loss/SRqqt1/assignment_loss‚0.4495325065376561 2loss/SRqqt2/assignment_loss‚0.4787031359352613 0loss/FRt1/detection_loss‚0.43232258960649833 /loss/FBt2/detection_loss‚0.5574419199885927 &loss/total_loss‚6.9416202237930325 0loss/FRt2/assignment_loss‚0.5948470283973135 /loss/FBt1/detection_loss‚0.3784584027699659³ ’ ÚÒ}ù2‚(:ò úÅöÒâóË network.recv‚ 3.796341e+06% disk./.usageGB‚1169.3697204589844 disk.md0.out‚ 682.34375 disk.dm-1.in‚0( disk.nvme1c1n1.out‚45341.02001953125$ memory_percent‚97.31464385986328) disk./.usagePercent‚76.16803259788206 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2951.56640625* proc.memory.percent‚18.014931678771973 cpu‚25.694334036527284& disk.nvme1n1.out‚45341.02001953125 disk.md0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0c0n1.out‚97721.623046875$ disk.nvme0n1.out‚97721.623046875 proc.cpu.threads‚17 network.sent‚ 2.5959696e+07$ disk.dm-0.out‚126060.90673828125" disk.dm-1.out‚15576.7724609375 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375% proc.memory.availableMB‚ 439.96875äÕà>žú2‚(:“ úÅöÒµ„¶Ì gpu.0.gpu‚16.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.635' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.211666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Rt_¢~û2‚(:ó ‰ÆöÒ©òÞË cpu‚25.694732331144866 network.sent‚ 2.6009283e+07 disk.dm-1.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚97813.474609375& disk.nvme1n1.out‚45341.72314453125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 309.37109375 network.recv‚ 3.80135e+06% disk./.usageGB‚1169.3694458007812$ disk.dm-0.out‚126152.73486328125" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0c0n1.out‚97813.474609375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0" disk.dm-1.out‚15576.8193359375( disk.nvme1c1n1.out‚45341.72314453125 disk.md0.out‚ 682.65625$ memory_percent‚97.31488227844238' proc.memory.availableMB‚ 439.9296875# proc.memory.rssMB‚ 2951.56640625* proc.memory.percent‚18.014931678771973) disk./.usagePercent‚76.16801470775322 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375>Ižü2‚(:“ ‰ÆöÒÛ¯«Ì gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚273.095' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.03166666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W ±ªý2‚(:ô ˜ÆöÒÈŸéË proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚45342.62939453125' proc.memory.availableMB‚ 439.9296875* proc.memory.percent‚18.014931678771973 cpu‚25.69512893496315% disk./.usageGB‚1169.6715850830078 disk.nvme0n1.in‚ 309.37109375 disk.dm-0.in‚ 539.2421875! disk.dm-1.out‚15577.021484375 network.sent‚ 2.6014168e+07 network.recv‚ 3.805195e+06" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0c0n1.out‚97874.470703125$ disk.nvme0n1.out‚97874.470703125 disk.md0.out‚ 683.0234375$ disk.dm-0.out‚126213.62841796875) disk./.usagePercent‚76.18769484338107 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚45342.62939453125 disk.md0.in‚0 disk.dm-1.in‚0$ memory_percent‚97.31488227844238# proc.memory.rssMB‚ 2951.566406254Fs€þ2‚(:’ ˜ÆöÒø€²Ì gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚258.92' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚86.30666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â5Q54ÿ2/ _wandbruntime‚14672  _runtime‚14672—IÌ€3Æ _step‚1332 ' lr-AdamW/pg1‚5.5279301819759876e-05 ' lr-AdamW/pg2‚5.5279301819759876e-05 trainer/global_step‚16199 % _timestamp‚1.7845214985141716e+09 _runtime‚14672.089688332©¿Ñ3Ë _runtime‚14672.089688332 _step‚1332 ' lr-AdamW/pg1‚5.5279301819759876e-05 ' lr-AdamW/pg2‚5.5279301819759876e-05 trainer/global_step‚16199 % _timestamp‚1.7845214985141716e+09´ R›Ék4‚3/ _wandbruntime‚14672  _runtime‚14672´Ús]Gƒ3…P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160284312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18789454510557 _step‚1333 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131162823 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66919103 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚538551770376704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚538551751764480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚16199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291447135 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18818183536640 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2479776281843200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291457280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2465262599470576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160294455 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160294455 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28261053 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2479776264803840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131162823 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28261056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38657761 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18817842112512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18923498752000 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitorOtÄ.on_train_batch_start/inactive_split.small_pool.current‚286 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2479776281843200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚557475376495104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18789795071831 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557475250516480 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2484052411581767 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚14672.09406866 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18817842112512 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2479776264803840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7845214985148802e+09 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131162825 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2498594465379840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131162825 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2498594106916352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38658047Œ,do(„3ŠP HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18818183536640 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2479776281843200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291457280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2465262599470576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160294455 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160294455 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28261053 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2479776264803840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131162823 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28261056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38657761 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18817842112512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18923498752000 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2479776281843200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚557475376495104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18789795071831 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557475250516480 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2484052411581767 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚14672.09406866 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18817842112512 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2479776264803840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7845214985148802e+09 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131162825 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2498594465379840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131162825 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2498594106916352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38658047 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160284312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18789454510557 _step‚1333 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131162823 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66919103 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚538551770376704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚538551751764480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 trainer/global_step‚16199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291447135 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128µ hKâp4…3/ _wandbruntime‚14672  _runtime‚14672¥õm9'†3N YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18790562898809 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18819293642752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18818952211456 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18924717388800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2479924556345344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28262790 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18790903460195 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160304125 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚538584932034560 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2465410030138048 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18818952211456 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131170618 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131170616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚306 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291474743 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2498743849988096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291464584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2465410047177408 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18819293642752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28262787 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 % _timestamp‚1.7845214991376662e+09 _step‚1334 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2498743849988096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚538584950646784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131170616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291474743 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291464584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2498743491517440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚557509668035584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66922852 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚303 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2479924539305984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131170618 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2479924539305984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131170618 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2479924556345344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28262790 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18790903460195 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160304125 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚538584932034560 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2465410030138048 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18818952211456 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131170618 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131170616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚306 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291474743 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2498743849988096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291464584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2465410047177408 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18819293642752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28262787 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 % _timestamp‚1.7845214991376662e+09 _step‚1334 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2498743849988096 SEDeviceStatsMoni¾pÂnZ tor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚538584950646784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131170616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291474743 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291464584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2498743491517440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚557509668035584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66922852¶ èù‡Ã4ˆ3/ _wandbruntime‚14672  _runtime‚14672WM[ì‰3‹ 0loss/FRt1/assignment_loss‚1.4979799208769284 3loss/FBt1/assignment_loss‚0.0021165865832014886 %loss/total_loss‚6.936507095710615 epoch‚40 _runtime‚14672.717034402 0loss/FRt2/assignment_loss‚0.6049062540463361 1loss/SRqqt1/detection_loss‚0.6371511171166013 .loss/FBt1/detection_loss‚0.384518773143322 0loss/FBt2/detection_loss‚0.39414624716102314 _step‚1335 2loss/SRqqt1/assignment_loss‚0.4220961375230179 3loss/SRqqt2/assignment_loss‚0.39724124376176756 4loss/FBt2/assignment_loss‚4.6751633291011984e-05 /loss/FRt1/detection_loss‚0.5019982243222595 % _timestamp‚1.7845214991387565e+09 /loss/FRt2/detection_loss‚1.4699459097588707 1loss/SRqqt2/detection_loss‚0.6243599297839967 trainer/global_step‚16199Úª]&Š3 trainer/global_step‚16199 0loss/FRt1/assignment_loss‚1.4979799208769284 3loss/FBt1/assignment_loss‚0.0021165865832014886 %loss/total_loss‚6.936507095710615 epoch‚40 _runtime‚14672.717034402 0loss/FRt2/assignment_loss‚0.6049062540463361 1loss/SRqqt1/detection_loss‚0.6371511171166013 .loss/FBt1/detection_loss‚0.384518773143322 0loss/FBt2/detection_loss‚0.39414624716102314 _step‚1335 2loss/SRqqt1/assignment_loss‚0.4220961375230179 3loss/SRqqt2/assignment_loss‚0.39724124376176756 4loss/FBt2/assignment_loss‚4.6751633291011984e-05 /loss/FRt1/detection_loss‚0.5019982243222595 % _timestamp‚1.7845214991387565e+09 /loss/FRt2/detection_loss‚1.4699459097588707 1loss/SRqqt2/detection_loss‚0.6243599297839967· õ»Pƒ‹3‚(:ø §ÆöÒÓèïË! disk.dm-1.out‚15577.126953125' disk.nvme0c0n1.out‚97956.1513671875# proc.memory.rssMB‚ 2951.56640625 proc.cpu.threads‚17 network.recv‚ 3.808297e+06% disk.nvme0n1.out‚97956.1513671875( disk.nvme1c1n1.out‚45364.27490234375 disk.md0.out‚ 683.41015625 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 439.8671875% disk./.usageGB‚1169.3769798278809 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375$ disk.dm-0.out‚126316.01904296875" disk.nvme0c0n1.in‚ 309.37109375* proc.memory.percent‚18.014931678771973 cpu‚25.695541843627616" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚45364.27490234375 disk.dm-1.in‚0$ memory_percent‚97.31526374816895 network.sent‚ 2.6030034e+07) disk./.usagePercent‚76.16850544392645 disk.md0.in‚0;4æPŒ3‚(:’ §ÆöÒü°Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.066' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.022000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó¬ „3‚(:ù ¶ÆöÒôøµË& disk.nvme1n1.out‚45402.87646484375 proc.cpu.threads‚17 disk.rbd0.out‚0 network.recv‚ 3.814445e+06) disk./.usagePercent‚76.16806688729568 disk.md0.out‚ 683.72265625" disk.dm-0.out‚126413.744140625" disk.dm-1.out‚15577.2587890625" disk.nvme1c1n1.in‚ 229.87109375* proc.memory.percent‚18.014931678771973 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0& disk.nvme0n1.out‚98016.09619140625( disk.nvme1c1n1.out‚45402.87646484375$ memory_percent‚97.31523990631104( proc.memory.availableMB‚ 439.87109375$ disk./.usageGB‚1169.370246887207 disk.md0.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2951.56640625 cpu‚25.695938768019925 network.sent‚ 2.6081657e+07" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚98016.09619140625 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375@”Ù¢žŽ3‚(:“ ¶ÆöÒ–ôóË gpu.0.gpu‚43.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚289.895' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.63166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xuˆ„3‚(:ù ÅÆöÒÈÞÔË disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚98075.40380859375( disk.nvme1c1n1.out‚45403.57861328125& disk.nvme1n1.out‚45403.57861328125" disk.dm-1.out‚15577.3876953125 network.recv‚ 3.817688e+06) disk./.usagePercent‚76.17245170818131% disk./.usageGB‚1169.4375648498535( disk.nvme0c0n1.out‚98075.40380859375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 684.16796875 disk.dm-1.in‚0 network.sent‚ 2.6086547e+07" disk.nvme0c0n1.in‚ 309.37109375 disk.dm-0.in‚ 539.2421875" disk.dm-0.out‚126472.673828125$ memory_percent‚97.31523990631104# proc.memory.rssMB‚ 2951.56640625 cpu‚25.69633235267802 proc.cpu.threads‚17 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 439.87109375* proc.memory.percent‚18.014931678771973&³ªž3‚(:“ ÅÆöÒѾšÌ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚269.437' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.81233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05P4‘3/ _wandbruntime‚14716  _runtime‚147166dÉ’3à _step‚1336 & lr-AdamW/pg1‚5.387833846307342e-05 & lr-AdamW/pg2‚5.387833846307342e-05 trainer/global_step‚16249 $ _timestamp‚1.784521542764021e+09 _runtime‚14716.339655008œI¸Î“3È trainer/global_step‚16249 $ _timestamp‚1.784521542764021e+09 _runtime‚14716.339655008 _step‚1336 & lr-AdamW/pg1‚5.387833846307342e-05 & lr-AdamW/pg2‚5.387833846307342e-05¸ G¦4”3/ _wandbruntime‚14716  _runtime‚14716cW\á (•3…P eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚540210783880704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131552473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160768964 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67120402 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚540210765268480 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131552475 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2506063759588864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18873411214336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38772374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2487190006950400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2506063401125376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚559190220307456 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2491479428168775 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2472634149883536 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131552473 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160779107 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28347753 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚14716.34063825 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131552475 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38772646 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18844937723965 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18873752638464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2487189989911040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160768964 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292321437 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2506063401125376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _step‚1337 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2487189989911040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67120127 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18873411214336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128U³Û|(–3ŠP F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292321437 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2506063401125376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _step‚1337 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2487189989911040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67120127 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18873411214336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚540210783880704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131552473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160768964 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67120402 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚540210765268480 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131552475 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2506063759588864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18873411214336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38772374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2487190006950400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2506063401125376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚559190220307456 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2491479428168775 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2472634149883536 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131552473 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160779107 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28347753 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _runtime‚14716.34063825 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131552475 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38772646 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18844937723965 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18873752638464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2487189989911040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160768964 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452¹ ³”4—3/ _wandbruntime‚14716  _runtime‚14716áÉl§ ˜3N % _timestamp‚1.7845215433876567e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160788777 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚540243964150784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚16249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚559224394632704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2506212785726464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18874862744576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38775000 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131560266 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18874521313280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160778620 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1338 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28349490 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18980554356224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292349045 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2487338264413184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131560266 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38774713 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚559224518507008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67124490 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2472781563511648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131560268 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2506213144197120 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18846046112217 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67124200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28349487 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2491627609623865 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160778620 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚540243945538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2472781580551008 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292349045 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18980449094144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18846386673603 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2487338281452544 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18874521313280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160788777 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2491627967224611 _runtime‚14716.963400629 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2472781563511648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131560268 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2506213144197120 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18846046112217 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67124200 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28349487 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2491627609623865 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160778620 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚540243945538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2472781580551008 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292349045 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18980449094144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18846386673603 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2487338281452544 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18874521313280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160788777 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2491627967224611 _runtime‚14716.963400629 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚16249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚559224394632704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2506212785726464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18874862744576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38775000 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131560266 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18874521313280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160778620 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1338 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28349490 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18980554356224 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292349045 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2487338264413184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131560266 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38774713 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚559224518507008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67124490º 3¤ñ24š3/ _wandbruntime‚14716  _runtime‚14716ÞÂé ›3‡ 0loss/FRt2/assignment_loss‚0.5954056878409317 3loss/SRqqt1/assignment_loss‚0.44279807034213603 4loss/FBt2/assignment_loss‚1.6384505439412886e-05 0loss/FRt1/detection_loss‚0.48348417348292116 1loss/SRqqt1/detection_loss‚0.5866605153577857 /loss/FRt2/detection_loss‚1.5273086900714758 $loss/total_loss‚6.87680497161308 trainer/global_step‚16249 % _timestamp‚1.7845215433886647e+09 0loss/FRt1/assignment_loss‚1.4271305206348188 2loss/FBt1/assignment_loss‚0.002687551212788059 _runtime‚ 14716.9639084 _step‚1339 2loss/SRqqt2/assignment_loss‚0.5128951678632513 1loss/SRqqt2/detection_loss‚0.6137405758665755 0loss/FBt1/detection_loss‚0.31029375924221575 -loss/FBt2/detection_loss‚0.37438387519274 epoch‚40êõÂbœ3Œ epoch‚40 0loss/FRt2/assignment_loss‚0.5954056878409317 3loss/SRqqt1/assignment_loss‚0.44279807034213603 4loss/FBt2/assignment_loss‚1.6384505439412886e-05 0loss/FRt1/detection_loss‚0.48348417348292116 1loss/SRqqt1/detection_loss‚0.5866605153577857 /loss/FRt2/detection_loss‚1.5273086900714758 $loss/total_loss‚6.87680497161308 trainer/global_step‚16249 % _timestamp‚1.7845215433886647e+09 0loss/FRt1/assignment_loss‚1.4271305206348188 2loss/FBt1/assignment_loss‚0.002687551212788059 _runtime‚ 14716.9639084 _step‚1339 2loss/SRqqt2/assignment_loss‚0.5128951678632513 1loss/SRqqt2/detection_loss‚0.6137405758665755 0loss/FBt1/detection_loss‚0.31029375924221575 -loss/FBt2/detection_loss‚0.37438387519274» &óì@„3‚(:ù ÔÆöÒÞúÉË" disk.dm-1.out‚15577.6435546875 disk.rbd0.out‚0$ memory_percent‚97.30064868927002 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 309.37109375# proc.memory.rssMB‚ 2951.56640625 cpu‚25.696726329383512 network.recv‚ 3.82118e+06% disk.nvme0n1.out‚98105.6787109375( disk.nvme1c1n1.out‚45408.51708984375 disk.nvme1n1.in‚ 229.87109375 disk.rbd0.in‚0( proc.memory.availableMB‚ 442.26171875* proc.memory.percent‚18.014931678771973% disk./.usageGB‚1169.4689903259277' disk.nvme0c0n1.out‚98105.6787109375 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚45408.51708984375 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚126506.93798828125 disk.dm-1.in‚0 network.sent‚ 2.6102413e+07) disk./.usagePercent‚76.17449863709022 disk.md0.in‚0 disk.md0.out‚ 684.48828125EŠp œž3‚(:‘ ÔÆöÒòúÌ gpu.0.gpu‚2.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.584' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.86133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 J•FƒŸ3‚(:ø ãÆöÒ°¤°Ë( proc.memory.availableMB‚ 442.31640625# proc.memory.rssMB‚ 2951.56640625 cpu‚25.697170727630603) disk./.usagePercent‚76.17903949965523* proc.memory.percent‚18.014931678771973 proc.cpu.threads‚17 network.sent‚ 2.6154026e+07 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚45477.44091796875 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚45477.44091796875$ disk./.usageGB‚1169.538703918457 disk.dm-0.in‚ 539.2421875" disk.dm-1.out‚15578.4873046875 disk.rbd0.out‚0% disk.nvme0n1.out‚98282.6376953125" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 684.9609375$ memory_percent‚97.30031490325928 network.recv‚ 3.825916e+06$ disk.dm-0.out‚126750.94970703125' disk.nvme0c0n1.out‚98282.6376953125 disk.md0.in‚0Ã?­@ž 3‚(:“ ãÆöÒÚ‘ïË gpu.0.gpu‚43.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.298' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.76599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹¶+¥€¡3‚(:õ òÆöÒÊß¹Ë proc.cpu.threads‚17 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0# proc.memory.rssMB‚ 2951.56640625* proc.memory.percent‚18.014931678771973 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0' disk.nvme0c0n1.out‚98340.6142578125% disk.nvme0n1.out‚98340.6142578125& disk.nvme1c1n1.out‚45480.947265625 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚45480.947265625' proc.memory.availableMB‚ 442.3203125 network.sent‚ 2.6158915e+07 network.recv‚ 3.829392e+06$ disk.dm-0.out‚126811.35693359375 disk.dm-1.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.md0.out‚ 685.38671875$ memory_percent‚97.30029106140137) disk./.usagePercent‚75.99420837889352$ disk./.usageGB‚1166.701084136963# disk.dm-1.out‚15578.64306640625 cpu‚25.697376354074795¥´Û;ž¢3‚(:“ òÆöÒô¤ýË gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.071' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.35700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞW4£3/ _wandbruntime‚14760  _runtime‚14760·ëã7ɤ3à & lr-AdamW/pg1‚5.249364835503724e-05 & lr-AdamW/pg2‚5.249364835503724e-05 trainer/global_step‚16299 $ _timestamp‚1.784521587185074e+09 _runtime‚14760.764109579 _step‚1340?\3PÎ¥3È trainer/global_step‚16299 $ _timestamp‚1.784521587185074e+09 _runtime‚14760.764109579 _step‚1340 & lr-AdamW/pg1‚5.249364835503724e-05 & lr-AdamW/pg2‚5.249364835503724e-05¼ Yïp@4¦3/ _wandbruntime‚14760  _runtime‚14760CYhdÉ&§3†P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18929321740288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18900420937373 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2513533053797888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2494603732057600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18900761498647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2498906444755783 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131942125 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2494603715018240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161263759 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67322605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67322302 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2494603732057600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18928980316160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38888149 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚560905054720000 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161253616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2480005683257136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚541869778772480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 293205884 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2513532695334400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2498906444755783 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131942125 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2494603715018240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161263759 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67322605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67322302 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2494603732057600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18928980316160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38888149 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚560905054720000 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161253616 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2480005683257136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚541869778772480 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 293205884 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2513532695334400 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18900420937373 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2513533053797888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2494603732057600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18900761498647 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2513682438406144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2499054983811619 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161273429 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38890373 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28436187 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293223347 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67326275 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2480153113924608 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131949916 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2499054626210873 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2494751989520384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2494752006559744 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2494752006559744 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19036274206208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2513682438406144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚560939348734464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293223347 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161263272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚560939233248768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚541902977654784 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38890088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18930431846400 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131949918 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224c*'«3’N >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2513682438406144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2499054983811619 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161273429 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38890373 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28436187 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293223347 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67326275 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2480153113924608 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131949916 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2499054626210873 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2494751989520384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2494752006559744 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2494752006559744 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19036274206208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2513682438406144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚560939348734464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293223347 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161263272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚560939233248768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚541902977654784 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38890088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚18930431846400 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131949918 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _step‚1342¾ ë>ã4¬3/ _wandbruntime‚14761  _runtime‚14761m– L­3ˆ epoch‚40 trainer/global_step‚16299 _step‚1343 2loss/SRqqt2/assignment_loss‚0.4551058555285793 2loss/FBt1/assignment_loss‚0.010001591342270189 0loss/FBt2/detection_loss‚0.40416419169577283 _runtime‚14761.387865052 2loss/SRqqt1/assignment_loss‚0.4358009285089896 /loss/FRt2/detection_loss‚1.5858156199090454 1loss/SRqqt1/detection_loss‚0.6558769315848814 /loss/FBt1/detection_loss‚0.4504749591517014 % _timestamp‚1.7845215878125615e+09 0loss/FRt1/assignment_loss‚1.5176278504681788 2loss/FBt2/assignment_loss‚0.004451895340477867 1loss/SRqqt2/detection_loss‚0.6397366099004661 0loss/FRt2/assignment_loss‚0.7224461140342283 /loss/FRt1/detection_loss‚0.4429055164290532 %loss/total_loss‚7.324408063893644¤Cù@®3 2loss/FBt2/assignment_loss‚0.004451895340477867 1loss/SRqqt2/detection_loss‚0.6397366099004661 0loss/FRt2/assignment_loss‚0.7224461140342283 /loss/FRt1/detection_loss‚0.4429055164290532 %loss/total_loss‚7.324408063893644 epoch‚40 trainer/global_step‚16299 _step‚1343 2loss/SRqqt2/assignment_loss‚0.4551058555285793 2loss/FBt1/assignment_loss‚0.010001591342270189 0loss/FBt2/detection_loss‚0.40416419169577283 _runtime‚14761.387865052 2loss/SRqqt1/assignment_loss‚0.4358009285089896 /loss/FRt2/detection_loss‚1.5858156199090454 1loss/SRqqt1/detection_loss‚0.6558769315848814 /loss/FBt1/detection_loss‚0.4504749591517014 % _timestamp‚1.7845215878125615e+09 0loss/FRt1/assignment_loss‚1.5176278504681788¿ à§s¹¯3‚(:ö ÇöÒéß™Ë network.sent‚ 2.6161837e+07) disk./.usagePercent‚76.00205444276304% disk./.usageGB‚1166.8215408325195 disk.rbd0.in‚0' disk.nvme0c0n1.out‚98783.4228515625 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375" disk.nvme0c0n1.in‚ 309.37109375% disk.nvme0n1.out‚98783.4228515625& disk.nvme1c1n1.out‚45605.697265625 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0# diP1ÇÈsk.dm-1.out‚15578.81103515625$ memory_percent‚97.30050563812256# proc.memory.rssMB‚ 2951.56640625 network.recv‚ 3.832581e+06 disk.md0.out‚ 685.8359375$ disk.dm-0.out‚127377.77099609375( proc.memory.availableMB‚ 442.28515625* proc.memory.percent‚18.014931678771973 cpu‚25.697177108257613 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚45605.697265625 disk.md0.in‚0s"‘~‘°3‚(:† ÇöÒÀ¨Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.093' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.031 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©êçé±3‚(:ö ÇöÒìÁûË( disk.nvme0c0n1.out‚98807.59912109375$ disk.nvme1n1.out‚45607.994140625 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0$ memory_percent‚97.30069637298584 proc.cpu.threads‚17 network.sent‚ 2.6226476e+07" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚98807.59912109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 686.078125# disk.dm-1.out‚15579.05224609375 network.recv‚ 3.838012e+06& disk.nvme1c1n1.out‚45607.994140625# disk.dm-0.out‚127403.4697265625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2951.56640625* proc.memory.percent‚18.014931678771973) disk./.usagePercent‚75.98659712296514% disk./.usageGB‚1166.5842323303223" disk.nvme1c1n1.in‚ 229.87109375 disk.rbd0.out‚0( proc.memory.availableMB‚ 442.25390625 cpu‚25.697244744655144–ÛŸ²3‚(:” ÇöÒµèºË gpu.0.gpu‚91.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚122.621' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.873666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mCº4³3/ _wandbruntime‚14803  _runtime‚14803¢îæ†É´3à & lr-AdamW/pg1‚5.112532043170975e-05 & lr-AdamW/pg2‚5.112532043170975e-05 trainer/global_step‚16349 $ _timestamp‚1.784521630055849e+09 _runtime‚14803.631564931 _step‚1344†kε3È & lr-AdamW/pg1‚5.112532043170975e-05 & lr-AdamW/pg2‚5.112532043170975e-05 trainer/global_step‚16349 $ _timestamp‚1.784521630055849e+09 _runtime‚14803.631564931 _step‚1344À H\dî4¶3/ _wandbruntime‚14803  _runtime‚14803œ-l (·3†P 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _step‚1345 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2521002348006912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19091137594880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132331775 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161748411 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28521153 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚14803.633544514 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2502017457164800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18984890842112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132331775 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39003265 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2502017440125440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294080186 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚543528792276480 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19091030228480 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2487377199591376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845216300579047e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161738268 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2521001989543424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚543528810888704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18956244712055 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18984549417984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39002991 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2487377216630736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2521002348006912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132331773 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67524421 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2506333461342791 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18955904150781 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67524144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2502017457164800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161738268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2506333103742157 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2521001989543424 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18984549417984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 trainer/global_step‚16349 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28521156 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚562619948483584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161748411 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18984890842112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2502017457164800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚18984890842112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132331775 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39003265 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2502017440125440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294080186 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚543528792276480 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19091030228480 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2487377199591376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845216300579047e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161738268 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2521001989543424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚543528810888704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚18956244712055 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚18984549417984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39002991 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2487377216630736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2521002348006912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132331773 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67524421 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2506333461342791 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚18955904150781 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67524144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2502017457164800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161738268 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2506333103742157 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2521001989543424 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚18984549417984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 trainer/global_step‚16349 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28521156 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚562619948483584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161748411 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚18984890842112Á TŽå4¹3/ _wandbruntime‚14804  _runtime‚14804Û°eã$º3N _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2502165731666944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67528214 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39005601 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132339568 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18985659516928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚14804.244386733 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132339566 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2506482000398627 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67528491 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294097649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39005327 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚562654120360448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2502165714627584 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132339566 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2487524630258848 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18985659516928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132339568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18957012539033 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2502165731666944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28522890 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18986000948224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2521151732615168 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161758081 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294087490 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2521151374144512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2487524647298208 trainer/global_step‚16349 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161747924 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18957353100419 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonië ?%2tor.on_train_batch_end/active.all.allocated‚ 294097649 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28522887 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2521151732615168 _step‚1346 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161747924 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚562654237943296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0…ì¸Ð'»3”N ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294097649 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39005327 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚562654120360448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2502165714627584 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132339566 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2487524630258848 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚18985659516928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132339568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚18957012539033 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2502165731666944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28522890 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚18986000948224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2521151732615168 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161758081 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294087490 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2521151374144512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2487524647298208 trainer/global_step‚16349 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161747924 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚18957353100419 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294097649 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28522887 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2521151732615168 _step‚1346 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 161747924 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚562654237943296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2502165731666944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67528214 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39005601 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132339568 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚18985659516928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚14804.244386733 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132339566 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2506482000398627 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67528491 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 -©±4¼3/ _wandbruntime‚14804  _runtime‚14804gštß½3ˆ % _timestamp‚1.7845216306711562e+09 _runtime‚14804.246484666 0loss/FRt2/assignment_loss‚0.6547078359491971 trainer/global_step‚16349 /loss/FRt1/detection_loss‚0.4692136790504696 /loss/FBt1/detection_loss‚0.4022730171685492 2loss/SRqqt2/assignment_loss‚0.4816048369452399 4loss/FBt1/assignment_loss‚0.00028382917709179103 2loss/FBt2/assignment_loss‚0.023610679555169156 /loss/FRt2/detection_loss‚1.5507657484321293 1loss/SRqqt2/detection_loss‚0.6017632483986121 /loss/FBt2/detection_loss‚0.2918427079178703 epoch‚40 _step‚1347 0loss/FRt1/assignment_loss‚1.5423201761446885 2loss/SRqqt1/assignment_loss‚0.4145194641677745 1loss/SRqqt1/detection_loss‚0.6297136558824894 $loss/total_loss‚7.06261887878928ÑÕ'%¾3 /loss/FBt2/detection_loss‚0.2918427079178703 epoch‚40 _step‚1347 0loss/FRt1/assignment_loss‚1.5423201761446885 2loss/SRqqt1/assignment_loss‚0.4145194641677745 1loss/SRqqt1/detection_loss‚0.6297136558824894 $loss/total_loss‚7.06261887878928 % _timestamp‚1.7845216306711562e+09 _runtime‚14804.246484666 0loss/FRt2/assignment_loss‚0.6547078359491971 trainer/global_step‚16349 /loss/FRt1/detection_loss‚0.4692136790504696 /loss/FBt1/detection_loss‚0.4022730171685492 2loss/SRqqt2/assignment_loss‚0.4816048369452399 4loss/FBt1/assignment_loss‚0.00028382917709179103 2loss/FBt2/assignment_loss‚0.023610679555169156 /loss/FRt2/detection_loss‚1.5507657484321293 1loss/SRqqt2/detection_loss‚0.6017632483986121à ^(r§~¿3‚(:ó ŸÇöÒ³ÞË) disk./.usagePercent‚75.98661650393805 disk.dm-1.in‚0 disk.rbd0.out‚0# disk.nvme0n1.out‚98808.43359375" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚45608.34814453125( proc.memory.availableMB‚ 441.76953125 network.recv‚ 3.841262e+06$ disk./.usageGB‚1166.584529876709 disk.md0.in‚0 disk.md0.out‚ 686.26171875 disk.dm-0.in‚ 539.2421875% disk.nvme0c0n1.out‚98808.43359375 disk.nvme0n1.in‚ 309.37109375# memory_percent‚97.3036527633667 disk.nvme1n1.in‚ 229.87109375# disk.dm-0.out‚127404.1962890625# disk.dm-1.out‚15579.10302734375" disk.nvme0c0n1.in‚ 309.37109375* proc.memory.percent‚18.014931678771973 cpu‚25.697400166023712 proc.cpu.threads‚17& disk.nvme1n1.out‚45608.34814453125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2951.56640625 network.sent‚ 2.622944e+07 =–êžÀ3‚(:“ ŸÇöÒ¿£Ì gpu.0.gpu‚92.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.972' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.99066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡¹ ;‚Á3‚(:÷ ®ÇöÒ†‡¹Ë disk.md0.out‚ 686.7109375( proc.memory.availableMB‚ 441.32421875# proc.memory.rssMB‚ 2951.56640625* proc.memory.percent‚18.014931678771973 network.recv‚ 3.846717e+06 disk.dm-1.in‚0( disk.nvme1c1n1.out‚47526.47216796875 proc.cpu.threads‚17( disk.nvme0c0n1.out‚98833.07666015625 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚98833.07666015625" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.30637073516846 cpu‚25.697572353286105( disk./.usagePercent‚76.1664831139447 disk.dm-0.in‚ 539.2421875# disk.dm-0.out‚129345.8291015625! disk.dm-1.out‚15579.287109375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme1n1.out‚47526.47216796875 disk.md0.in‚0 network.sent‚ 2.6293821e+07$ disk./.usageGB‚1169.345932006836 disk.rbd0.in‚0 disk.nvme1n1.in‚ 229.871093751আ“Â3‚(:ˆ ®ÇöÒØžƒÌ gpu.0.gpu‚71.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.461' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.487 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ›w¢}Ã3‚(:ò ½ÇöÒûýÈË& disk.nvme0n1.out‚98834.29443359375" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚47527.17431640625 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875( disk.nvme0c0n1.out‚98834.29443359375 network.sent‚ 2.6296844e+07$ disk./.usageGB‚1169.345989227295 disk.md0.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚15579.380859375 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2951.56640625& disk.nvme1n1.out‚47527.17431640625 disk.md0.out‚ 687.21875# disk.dm-0.out‚129346.5712890625* proc.memory.percent‚18.014931678771973 cpu‚25.697439674672445 disk.nvme0n1.in‚ 309.37109375" disk.nvme0c0n1.in‚ 309.37109375$ memory_percent‚97.30806350708008& proc.memory.availableMB‚ 441.046875 proc.cpu.threads‚17 network.recv‚ 3.8499e+06) disk./.usagePercent‚76.16648684105488;Tn8žÄ3‚(:“ ½ÇöÒÏñ×Ë gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚273.196' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.06533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ónÇ4Å3/ _wandbruntime‚14846  _runtime‚14846HZ=ÌÆ3Æ % _timestamp‚1.7845216730180252e+09 _runtime‚14846.593684823 _step‚1348 ' lr-AdamW/pg1‚4.9773442578236394e-05 ' lr-AdamW/pg2‚4.9773442578236394e-05 trainer/global_step‚16399b`,ëÑÇ3Ë ' lr-AdamW/pg1‚4.9773442578236394e-05 ' lr-AdamW/pg2‚4.9773442578236394e-05 trainer/global_step‚16399 % _timestamp‚1.7845216730180252e+09 _runtime‚14846.593684823 _step‚1348Ä sÜ{t4È3/ _wandbruntime‚14846  _runtime‚14846/çY (É3†P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2528471283752448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚564334802352640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2509431182272000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39119154 trainer/global_step‚16399 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19040459943936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162233063 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19146977959936 _step‚1349 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132721425 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132721423 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162222920 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2528471642215936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28607856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚564334670082560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19040118519808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2513760120329165 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2494748732964976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19011387364189 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2509431182272000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132721423 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2528471283752448 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294954488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39118855 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2528471283752448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚564334802352640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2509431182272000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39119154 trainer/global_step‚16399 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19040459943936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162233063 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19146977959936 _step‚1349 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132721425 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132721423 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162222920 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2528471642215936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28607856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚564334670082560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19040118519808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2513760120329165 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2494748732964976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19011387364189 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2509431182272000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132721423 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2528471283752448 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294954488 JDeviceStatsƒ ô†Monitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39118855 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294971951 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2494896163632448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2513908659384889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2528620668353536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67730777 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚545220986050560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294961792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132729218 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39121190 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2509579439734784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2494896180671808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2528621026824192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28609587 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19041228618752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294971951 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132729218 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162242733 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 trainer/global_step‚16399 _step‚1350 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39121486 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚564369096875008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19147978561536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2509579456774144 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2509579439734784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28609590 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚564368964612096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799ØØñ¢'Í3•N [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2494896163632448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2513908659384889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2528620668353536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67730777 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚545220986050560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294961792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132729218 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39121190 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2509579439734784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2494896180671808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2528621026824192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28609587 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19041228618752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294971951 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132729218 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162242733 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 trainer/global_step‚16399 _step‚1350 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39121486 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚564369096875008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19147978561536 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2509579456774144 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2509579439734784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28609590 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚564368964612096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _runtime‚14847.220354723 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294971951Æ ÒL³M4Î3/ _wandbruntime‚14847  _runtime‚14847.Uð»Ï3ˆ 0loss/FBt1/detection_loss‚0.40841884794102395 /loss/FBt2/detection_loss‚0.2025166394887492 trainer/global_step‚16399 % _timestamp‚1.7845216736463542e+09 _step‚1351 3loss/FBt2/assignment_loss‚0.0018384990147964687 2loss/FBt1/assignment_loss‚0.005341756419895385 .loss/FRt2/detection_loss‚1.487215134319617 0loss/FRt2/assignment_loss‚0.7304340283831054 3loss/SRqqt1/assignment_loss‚0.42730311511196567 2loss/SRqqt2/assignment_loss‚0.4622594731550191 /loss/FRt1/detection_loss‚0.4764247065471298 0loss/SRqqt2/detection_loss‚0.607741927280965 %loss/total_loss‚6.936751992532806 epoch‚40 _runtime‚14847.221672395 0loss/FRt1/assignment_loss‚1.4701288399433998 1loss/SRqqt1/detection_loss‚0.6571290249271395ÿÂÞÐ3 2loss/FBt1/assignment_loss‚0.005341756419895385 .loss/FRt2/detection_loss‚1.487215134319617 0loss/FRt2/assignment_loss‚0.7304340283831054 3loss/SRqqt1/assignment_loss‚0.42730311511196567 2loss/SRqqt2/assignment_loss‚0.4622594731550191 /loss/FRt1/detection_loss‚0.4764247065471298 0loss/SRqqt2/detection_loss‚0.607741927280965 %loss/total_loss‚6.936751992532806 epoch‚40 _runtime‚14847.221672395 0loss/FRt1/assignment_loss‚1.4701288399433998 1loss/SRqqt1/detection_loss‚0.6571290249271395 0loss/FBt1/detection_loss‚0.40841884794102395 /loss/FBt2/detection_loss‚0.2025166394887492 trainer/global_step‚16399 % _timestamp‚1.7845216736463542e+09 _step‚1351 3loss/FBt2/assignment_loss‚0.0018384990147964687Ç 8Š.ÌÑ3‚(:ô ÌÇöÒø˜ºË# memory_percent‚97.3402738571167* proc.memory.percent‚18.047118186950684 cpu‚25.697896743548405 network.sent‚ 2.6301722e+07) disk./.usagePercent‚76.16347160892211% disk./.usageGB‚1169.2996978759766& disk.nvme1n1.out‚47759.91259765625 disk.md0.in‚0 disk.md0.out‚ 687.53125 disk.rbd0.in‚0( proc.memory.availableMB‚ 435.76953125 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375# proc.memory.rssMB‚ 2956.83984375 proc.cpu.threads‚17 network.recv‚ 3.853396e+06 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚130217.53564453125 disk.dm-1.out‚15579.45703125" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0c0n1.out‚99473.2880859375 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚47759.91259765625 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0% disk.nvme0n1.out‚99473.2880859375ÎzÊ‘Ò3‚(:† ÌÇöÒœÊÐÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.856' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ”Ðâ4Ó3/ _wandbruntime‚14863  _runtime‚14863R U¸/Ô3Æz 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5955660939216614 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07246309518814087 particle/accuracy_2_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 trainer/global_step‚16399 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2982601225376129 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07311046868562698 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.042050860822200775 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5946269035339355 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8516817688941956 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6539520621299744 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0608365535736084 _runtime‚14863.396884371 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5025030970573425 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.273214727640152 /particle/accuracy_3_of_3‚0.9994120597839355 $ _timestamp‚1.784521689820787e+09 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6336548924446106 *jet/accuracy_1_of_4‚0.9588440656661987 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 epoch‚40 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 )loss/val_total_loss‚7.403525725663274 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08130534738302231 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.347624272108078 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5758622884750366 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4574474096298218 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992322325706482 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9105682373046875 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29745185375213623 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _step‚1352 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26049283146858215 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6227283477783203 7"Purity/0FRt1SRqqt1FBt/event_purity‚0.58392333984375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_4‚0.7359887957572937 +particle/specificity‚0.9098973274230957 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22533738613128662 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2083548605442047 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8715105652809143 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_3‚0.07456566393375397 /particle/accuracy_4_of_4‚0.9565867185592651 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9971833229064941 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9907387495040894 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5653220415115356 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05438797175884247 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5572916269302368 jet/accuracy_1_of_5‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5955062508583069 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 particle/accuracy_1_of_2‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7125736474990845 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3427765369415283 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6897735595703125 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_4‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7047559022903442 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8876297473907471 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt0FBt/event_proportio§»dæ)n‚0.4408714771270752 *jet/accuracy_1_of_2‚0.7594624757766724 6validation_average_jet_accuracy‚0.3691941797733307 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5758622884750366 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6133333444595337 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21230924129486084 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6602605581283569 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4853682816028595 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6668347120285034 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28632214665412903 +jet/accuracy_2_of_2‚0.15965786576271057 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7041400671005249 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24288050830364227 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9967963695526123 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9903878569602966 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.685219407081604 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6475095748901367 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6782127022743225 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6360262632369995 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6467323899269104 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2972985804080963 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30057603120803833 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7345387935638428 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.23666667938232422 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_2_of_2‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.621183454990387 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03900386765599251 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.739111065864563 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8523491621017456 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6221920847892761 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.757036030292511 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7179471850395203 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_4_of_5‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22594065964221954 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7121697068214417 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4636944830417633 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5613086223602295 particle/accuracy_1_of_4‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04979328811168671 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3369084894657135 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.872826874256134 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8668745160102844 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06979232281446457 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1052580252289772 +particle/sensitivity‚0.6419351696968079 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47761666774749756 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7909108400344849 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27748435735702515 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9499731659889221 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 *jet/accuracy_1_of_3‚0.9175891876220703 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5829352140426636 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5778020024299622 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20248912274837494 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6178925037384033 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6851851940155029 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2820996344089508 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.256468802690506 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6221688985824585 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6447272300720215 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8815948963165283 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4845547378063202 +jet/accuracy_3_of_4‚0.08606330305337906 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26905855536460876 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21602332592010498 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7630208730697632 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2681456208229065 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_4_of_5‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25808581709861755 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5817133784294128 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 jet/accuracy_3_of_5‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30528998374938965 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21231842041015625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21531474590301514 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2902119755744934 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3425998389720917 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6851851940155029 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.32927069067955017 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07024708390235901 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5747308135032654 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7771720886230469 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6208772659301758 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.640717625617981 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28632214665412903 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7771720886230469 particle/accuracy_1_of_1‚1 jet/accuracy_2_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988906383514404 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0608365535736084 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6695165634155273 6"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5970498919487 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2858479917049408 'particle/f_score‚0.6512335538864136 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.855744481086731 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28615444898605347ŒójQ=Õ3Ëz =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47761666774749756 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7909108400344849 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27748435735702515 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9499731659889221 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 *jet/accuracy_1_of_3‚0.9175891876220703 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5829352140426636 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5778020024299622 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20248912274837494 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6178925037384033 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6851851940155029 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2820996344089508 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.256468802690506 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6221688985824585 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6447272300720215 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8815948963165283 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4845547378063202 +jet/accuracy_3_of_4‚0.08606330305337906 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.26905855536460876 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21602332592010498 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7630208730697632 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2681456208229065 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_4_of_5‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25808581709861755 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5817133784294128 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 jet/accuracy_3_of_5‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30528998374938965 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21231842041015625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21531474590301514 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2902119755744934 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3425998389720917 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6851851940155029 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.32927069067955017 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07024708390235901 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5747308135032654 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7771720886230469 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6208772659301758 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.640717625617981 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28632214665412903 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7771720886230469 particle/accuracy_1_of_1‚1 jet/accuracy_2_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988906383514404 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0608365535736084 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6695165634155273 6"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5970498919487 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2858479917049408 'particle/f_score‚0.6512335538864136 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.855744481086731 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28615444898605347 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5955660939216614 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07246309518814087 particle/accuracy_2_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 trainer/global_step‚16399 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2982601225376129 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07311046868562698 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.042050860822200775 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5946269035339355 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8516817688941956 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6539520621299744 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0608365535736084 _runtime‚14863.396884371 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5025030970573425 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.273214727640152 /particle/accuracy_3_of_3‚0.9994120597839355 $ _timestamp‚1.784521689820787e+09 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6336548924446106 *jet/accuracy_1_of_4‚0.9588440656661987 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 epoch‚40 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 )loss/val_total_loss‚7.403525725663274 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08130534738302231 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.347624272108078 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5758622884750366 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4574474096298218 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992322325706482 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9105682373046875 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29745185375213623 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _step‚1352 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26049283146858215 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6227283477783203 7"Purity/0FRt1SRqqt1FBt/event_purity‚0.58392333984375 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_4‚0.7359887957572937 +particle/specificity‚0.9098973274230957 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22533738613128662 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2083548605442047 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8715105652809143 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_3‚0.07456566393375397 /particle/accuracy_4_of_4‚0.9565867185592651 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9971833229064941 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9907387495040894 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5653220415115356 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05438797175884247 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5572916269302368 jet/accuracy_1_of_5‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5955062508583069 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 particle/accuracy_1_of_2‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7125736474990845 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3427765369415283 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6897735595703125 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_4‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7047559022903442 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8876297473907471 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_1_of_2‚0.7594624757766724 6validation_average_jet_accuracy‚0.3691941797733307 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5758622884750366 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6133333444595337 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21230924129486084 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6602605581283569 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4853682816028595 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6668347120285034 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28632214665412903 +jet/accuracy_2_of_2‚0.15965786576271057 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7041400671005249 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24288050830364227 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9967963695526123 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9903878569602966 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.685219407081604 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6475095748901367 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6782127022743225 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6360262632369995 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6467323899269104 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2972985804080963 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30057603120803833 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7345387935638428 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.23666667938232422 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_2_of_2‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.621183454990387 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03900386765599251 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.739111065864563 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8523491621017456 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6221920847892761 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.757036030292511 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7179471850395203 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_4_of_5‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22594065964221954 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7121697068214417 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4636944830417633 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5613086223602295 particle/accuracy_1_of_4‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04979328811168671 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3369084894657135 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.872826874256134 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8668745160102844 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06979232281446457 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1052580252289772 +particle/sensitivity‚0.6419351696968079 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1È ’õÚ\Ö3‚(:’ ÛÇöÒ•¡¶Ë gpu.0.gpu‚93.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚256.76' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.58666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ånUµv×3‚(:ë ÛÇöÒª€éË network.recv‚ 3.858462e+06) disk./.usagePercent‚76.21420826280647% disk./.usageGB‚1170.0786323547363 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 99720.1640625" disk.nvme0n1.out‚ 99720.1640625 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚47812.99169921875 network.sent‚ 2.6378316e+07 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0 disk.dm-1.out‚15579.53515625" disk.nvme0c0n1.in‚ 309.37109375# memory_percent‚97.4827527999878( proc.memory.availableMB‚ 412.42578125* proc.memory.percent‚18.199777603149414 disk.md0.out‚687.8125 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚47812.99169921875" proc.memory.rssMB‚ 2981.8515625 proc.cpu.threads‚17 disk.md0.in‚0$ disk.dm-0.out‚130516.79345703125 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375 cpu‚25.70021588679949¢¹È¹…Ø3‚(:ú êÇöÒÙÊÊË& disk.nvme1n1.out‚47842.00732421875 disk.md0.in‚0" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚47842.00732421875$ memory_percent‚97.48060703277588) disk./.usagePercent‚76.20998892561268% disk./.usageGB‚1170.0138549804688 disk.md0.out‚ 688.1171875 disk.dm-0.in‚ 539.2421875 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚100155.67236328125* proc.memory.percent‚18.199777603149414 proc.cpu.threads‚17 disk.nvme1n1.in‚ 229.87109375# disk.dm-0.out‚130980.4560546875! disk.dm-1.out‚15579.720703125( proc.memory.availableMB‚ 412.77734375 cpu‚25.699998348941193 network.sent‚ 2.6423126e+07 network.recv‚ 3.863475e+06 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) disk.nvme0c0n1.out‚100155.67236328125" proc.memory.rssMB‚ 2981.8515625ÓôÈžÙ3‚(:“ êÇöÒﶕÌ gpu.0.gpu‚92.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚273.599' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.19966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½D?Ú3‚(:ô ùÇöÒǺòË' disk.nvme0n1.out‚100157.25732421875! disk.dm-1.out‚15579.798828125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375$ memory_percent‚97.48058319091797% proc.memory.availableMB‚ 412.78125 network.sent‚ 2.6428018e+07) disk./.usagePercent‚76.21001103979972' disk.nvme1c1n1.out‚47842.4892578125$ disk.dm-0.out‚130981.77783203125 disk.rbd0.out‚0" proc.memory.rssMB‚ 2981.8515625* proc.memory.percent‚18.199777603149414 cpu‚25.69979604625709 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375% disk.nvme1n1.out‚47842.4892578125 disk.md0.in‚0 disk.dm-1.in‚0) disk.nvme0c0n1.out‚100157.25732421875 proc.cpu.threads‚17 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 688.421875 disk.dm-0.in‚ 539.2421875 network.recv‚ 3.866931e+06% disk./.usageGB‚1170.0141944885254²ÖÜAžÛ3‚(:“ ùÇöÒº¥ÀÌ gpu.0.gpu‚69.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚266.112' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.70400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡Éê]4Ü3/ _wandbruntime‚14906  _runtime‚14906™dºMÉÝ3à _step‚1353 & lr-AdamW/pg1‚4.843810162320543e-05 & lr-AdamW/pg2‚4.843810162320543e-05 trainer/global_step‚16449 % _timestamp‚1.7845217325950403e+09 _runtime‚14906.17067555×?äYÎÞ3È % _timestamp‚1.7845217325950403e+09 _runtime‚14906.17067555 _step‚1353 & lr-AdamW/pg1‚4.843810162320543e-05 & lr-AdamW/pg2‚4.843810162320543e-05 trainer/global_step‚16449É UÛØØ4ß3/ _wandbruntime‚14906  _runtime‚14906S¾îåà3…P ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2537903598136320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162789066 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚14906.175077626 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2518787251254784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162799209 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19223448197120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚1354 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2518787234215424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28698669 D;DeviceStatsMonitor.on_train_batch_start/segment.allÊ7Èb& .current‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19116346881536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162789066 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133233883 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚566092846144000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2518787251254784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 296033092 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚566092959539712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133233883 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28698672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2523128696773294 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546869416559104 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2504041539386160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67972583 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚546869397946880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19087174426494 % _timestamp‚1.7845217325960238e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19116346881536 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162799209 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133233881 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2518787234215424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19116005457408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39273911 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2523129054373928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2537903239672832 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19223542980608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67972308 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296022947 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133233881 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2537903598136320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19087514987768 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19116005457408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496´Ÿ˜[(á3ŠP H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2518787234215424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19116005457408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39273911 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2523129054373928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2537903239672832 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19223542980608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67972308 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296022947 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133233881 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2537903598136320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19087514987768 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19116005457408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2537903598136320 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 162789066 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚14906.175077626 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2518787251254784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162799209 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19223448197120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚1354 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2518787234215424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28698669 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19116346881536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162789066 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133233883 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚566092846144000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2518787251254784 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 296033092 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚566092959539712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133233883 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28698672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2523128696773294 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚546869416559104 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2504041539386160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67972583 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚546869397946880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19087174426494 % _timestamp‚1.7845217325960238e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19116346881536 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162799209 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133233881 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296050555 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162798722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2523277235829018 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19088623376132 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133241676 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162798722 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2518935508717568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2523277593429764 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133241674 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133241676 _runtime‚14906.789519147 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚566127259829248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2538052982744576 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67976295 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19224663000064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚1355 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2518935525756928 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚546902596829184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 296040396 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2518935508717568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19224663000064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚1355 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2518935525756928 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚546902596829184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 296040396 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2518935508717568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296050555 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162798722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2523277235829018 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19088623376132 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133241676 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162798722 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2518935508717568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2523277593429764 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133241674 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133241676 _runtime‚14906.789519147 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚566127259829248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2538052982744576 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67976295 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2545372533881856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68174450 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚14949.322178671 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2545372892345344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19171574559232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133623533 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39389078 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296897249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133623533 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚16499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28785369 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19142998201176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2526200976361984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2526200976361984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19171915983360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2526200959322624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163283861 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _step‚1358 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 296907394 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚548528411450880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19279396684288 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68174189 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163273718ÞS](ó3‹P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2526200976361984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19171915983360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2526200959322624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163283861 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _step‚1358 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 296907394 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚548528411450880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19279396684288 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68174189 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163273718 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚567807696574464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133623531 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2530556070960936 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19142657639902 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 % _timestamp‚1.7845217757454915e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2526200959322624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚567807826747392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163283861 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2545372533881856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68174450 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚14949.322178671 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2545372892345344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19171574559232 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133623533 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39389078 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296897249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133623533 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚16499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28785369 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19142998201176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2526200976361984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488Î 1'u4ô3/ _wandbruntime‚14949  _runtime‚14949Zþ¹pµõ3N B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784521776361814e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39391072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2511560486387872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133631326 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133631324 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚548561591720960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19280403814912 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2526349233824768 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2530704610016772 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2526349233824768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68178473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2530704252416026 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2545522276953600 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2526349250864128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133631324 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19173026089472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68178175 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2511560503427232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296924857 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚567842115215872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚548561610333184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19173026089472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28787103 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2545522276953600 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19172684658176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19144106589540 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28787106 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39391367 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚567841995535872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚14949.937633087 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163293531 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 296914698 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2526349250864128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2545521918482944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163283374 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68178175 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2511560503427232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296924857 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚567842115215872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚548561610333184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19173026089472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28787103 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2545522276953600 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19172684658176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19144106589540 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28787106 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39391367 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚567841995535872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚14949.937633087 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163293531 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 296914698 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2526349250864128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2545521918482944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163283374 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784521776361814e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39391072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2511560486387872Ï ÅŸð4÷3/ _wandbruntime‚14949  _runtime‚14949}P“…ø3Š 2loss/SRqqt2/assignment_loss‚0.4994697468942508 3loss/FBt1/assignment_loss‚1.800078569922258e-06 1loss/SRqqt2/detection_loss‚0.6418231613254951 epoch‚41 % _timestamp‚1.7845217763628728e+09 _runtime‚14949.938207198 /loss/FRt1/assignment_loss‚1.440846021885074 2loss/FBt2/assignment_loss‚9.68272165247086e-06 1loss/SRqqt1/detection_loss‚0.6589965925293696 0loss/FBt1/detection_loss‚0.38219720891925324 0loss/FRt2/assignment_loss‚0.6738942510338239 /loss/FRt2/detection_loss‚1.4508194571421031 %loss/total_loss‚6.994011862637205 trainer/global_step‚16499 3loss/SRqqt1/assignment_loss‚0.36884734530286645 0loss/FRt1/detection_loss‚0.44166462651400984 /loss/FBt2/detection_loss‚0.4354419682907367 _step‚1360K4Lù3 2loss/SRqqt2/assignment_loss‚0.4994697468942508 3loss/FBt1/assignment_loss‚1.800078569922258e-06 1loss/SRqqt2/detection_loss‚0.6418231613254951 epoch‚41 % _timestamp‚1.7845217763628728e+09 _runtime‚14949.938207198 /loss/FRt1/assignment_loss‚1.440846021885074 2loss/FBt2/assignment_loss‚9.68272165247086e-06 1loss/SRqqt1/detection_loss‚0.6589965925293696 0loss/FBt1/detection_loss‚0.38219720891925324 0loss/FRt2/assignment_loss‚0.6738942510338239 /loss/FRt2/detection_loss‚1.4508194571421031 %loss/total_loss‚6.994011862637205 trainer/global_step‚16499 3loss/SRqqt1/assignment_loss‚0.36884734530286645 0loss/FRt1/detection_loss‚0.44166462651400984 /loss/FBt2/detection_loss‚0.4354419682907367 _step‚1360Ð 6B±Â‚ú3‚(:÷ µÈöÒìŒÃË cpu‚25.698913202955588 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚100164.50439453125 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚47845.03466796875 disk.md0.in‚0 disk.md0.out‚ 689.7265625* proc.memory.percent‚18.299269676208496( disk./.usagePercent‚76.2099809744443 disk.dm-0.in‚ 539.2421875" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚47845.03466796875' proc.memory.availableMB‚ 442.5390625# proc.memory.rssMB‚ 2998.15234375 proc.cpu.threads‚17 network.sent‚ 2.650235e+07 network.recv‚ 3.881371e+06% disk./.usageGB‚1170.0137329101562" disk.dm-0.out‚130988.337890625 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-1.in‚0" disk.dm-1.out‚15580.1884765625) disk.nvme0c0n1.out‚100164.50439453125# memory_percent‚97.2989559173584ݪW¼žû3‚(:“ µÈöÒÈø¿Ì gpu.0.gpu‚91.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.318' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.77266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¸‚ü3‚(:÷ ÄÈöÒÆàÒË network.sent‚ 2.6565964e+07) disk./.usagePercent‚76.20998967103472( disk.nvme1c1n1.out‚47845.71826171875 disk.dm-1.in‚0" disk.dm-1.out‚15580.3095703125) disk.nvme0c0n1.out‚100165.48095703125" proc.memory.rssMB‚ 3007.3671875* proc.memory.percent‚18.355512619018555 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚47845.71826171875 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚97.2581148147583( proc.memory.availableMB‚ 449.23046875 proc.cpu.threads‚17 network.recv‚ 3.886757e+06% disk./.usageGB‚1170.0138664245605" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 690.140625 disk.dm-0.in‚ 539.2421875" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375" disk.dm-0.out‚130988.986328125' disk.nvme0n1.out‚100165.48095703125 cpu‚25.69869693746896ƒÔÉDžý3‚(:“ ÄÈöÒÓþ»Ì gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚124.243' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.41433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<û<{þ3‚(:ð ÓÈöÒâõ·Ë$ proc.memory.availableMB‚471.1875 proc.cpu.threads‚17% disk./.usageGB‚1170.0134468078613& disk.nvme1n1.out‚47846.80322265625 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚100167.779296875! proc.memory.rssMB‚ 3015.265625) proc.memory.percent‚18.40372085571289 network.sent‚ 2.6571654e+07% disk.nvme0n1.out‚100167.779296875 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚130990.77001953125 disk.rbd0.in‚0$ memory_percent‚97.12409973144531 cpu‚25.69849902778908 network.recv‚ 3.890167e+06 disk.nvme0n1.in‚ 309.37109375" disk.dm-1.out‚15580.3876953125) disk./.usagePercent‚76.20996233889342" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚47846.80322265625 disk.md0.out‚ 690.859375" disk.nvme0c0n1.in‚ 309.37109375« žÿ3‚(:“ ÓÈöÒìLJÌ gpu.0.gpu‚69.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.564' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.52133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛöÝ&4€4/ _wandbruntime‚14992  _runtime‚14992›Àø?Ê4Ä & lr-AdamW/pg1‚4.581737240664533e-05 & lr-AdamW/pg2‚4.581737240664533e-05 trainer/global_step‚16549 % _timestamp‚1.7845218190932527e+09 _runtime‚14992.668926095 _step‚1361QÎÌÏ‚4É & lr-AdamW/pg1‚4.581737240664533e-05 & lr-AdamW/pg2‚4.581737240664533e-05 trainer/global_step‚16549 % _timestamp‚1.7845218190932527e+09 _runtime‚14992.668926095 _step‚1361Ñ /äï£4ƒ4/ _wandbruntime‚14992  _runtime‚14992‚ÜE†c„4†P QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68375536 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚1362 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2533614701469184 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19335211273216 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163768513 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚569522535153152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚16549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163768513 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2533614684429824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19198481414584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297771551 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚550187424954880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68375251 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2518784606133360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19198140853310 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2533614684429824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2552842186554368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19227143661056 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2537982729947310 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytesN`ö© .all.allocated‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2552841828090880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19227485085184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19227485085184 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19227143661056éDTO(…4‹P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68375251 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2518784606133360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19198140853310 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2533614684429824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2552842186554368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19227143661056 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2537982729947310 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2552841828090880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19227485085184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19227485085184 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19227143661056 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68375536 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚1362 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2533614701469184 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19335211273216 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163768513 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚569522535153152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 trainer/global_step‚16549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163768513 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2533614684429824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19198481414584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297771551 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚550187424954880 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272Ò zf4†4/ _wandbruntime‚14993  _runtime‚14993ñpÅ'‡4N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28873806 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297789000 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 297789000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134020974 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2552991571162624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19228253760000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2533762958931968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚569556950839296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163768026 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39505466 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550220623837184 % _timestamp‚1.7845218197128987e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134020974 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68379269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2533762958931968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163778183 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2552991212691968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19228595191296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297799159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ú¨ó£Ú"ˆ4”N K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68379269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2533762958931968 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163778183 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2552991212691968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19228595191296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297799159 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28873806 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297789000 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 297789000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134020974 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2552991571162624 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19228253760000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2533762958931968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚569556950839296 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163768026 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39505466 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550220623837184 % _timestamp‚1.7845218197128987e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134020974 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19253964627992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 298655998 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134402831 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚551846457071104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19390958505984 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2526156139506960 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚571237396964864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164253165 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2560311480763392 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134402833 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298645853 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134402833 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2541028426576384 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2541028409537024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 % _timestamp‚1.7845218628081477e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164243022 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2560311480763392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19283054187008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19282712762880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164243022 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2526156122467600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚15036.383694688 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2541028426576384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19391072163840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2560311122299904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164253165 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68576819 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39617760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68576529 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2545409746534318 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28958769 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19283054187008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2541028409537024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19282712762880 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19253624066718 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚551846438458880 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134402831 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2560311122299904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚1366 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19253964627992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 298655998 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134402831 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚551846457071104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19390958505984 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2526156139506960 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚571237396964864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164253165 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2560311480763392 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134402833 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 298645853 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134402833 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2541028426576384 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2541028409537024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚290 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 % _timestamp‚1.7845218628081477e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164243022 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2560311480763392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19283054187008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19282712762880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164243022 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2526156122467600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚15036.383694688 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2541028426576384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19391072163840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2560311122299904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164253165 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68576819 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39617760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68576529 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2545409746534318 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28958769 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19283054187008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2541028409537024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19282712762880 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19253624066718 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚551846438458880 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134402831 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2560311122299904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚1366Ö b\é4˜4/ _wandbruntime‚15037  _runtime‚15037@–¸?>™4N YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19254732454970 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298673461 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164262835 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28960503 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚16599 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2526303553135072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19392072830976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68580602 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚551879637341184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2541176701078528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2541176684039168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134410624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232~7}š'š4•N FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68580602 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚551879637341184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2541176701078528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2541176684039168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134410624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19254732454970 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298673461 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164262835 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28960503 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚16599 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2526303553135072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19392072830976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 B»lƒž4‚(:ø ÉöÒöŒøË proc.cpu.threads‚17# disk.dm-1.out‚15580.82177734375& disk.nvme0n1.out‚100199.8876953125( disk.nvme1c1n1.out‚47942.80908203125' proc.memory.availableMB‚ 486.7578125 cpu‚25.69820441104377 network.recv‚ 3.905698e+06% disk./.usageGB‚1169.9467811584473 disk.md0.out‚ 692.0546875" disk.nvme0c0n1.in‚ 309.37109375* proc.memory.percent‚18.452048301696777) disk./.usagePercent‚76.20562000706383 disk.md0.in‚0" disk.dm-0.out‚131115.830078125 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚100199.8876953125 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 network.sent‚ 2.6660086e+07 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚47942.80908203125$ memory_percent‚97.02906608581543# proc.memory.rssMB‚ 3023.18359375_³i‘Ÿ4‚(:† ÉöÒÃã³Ì gpu.0.gpu‚42.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.41' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.47 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,=Ží‚ 4‚(:÷ žÉöÒû®¢Ë" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375$ disk.nvme0n1.out‚100203.18359375 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚47964.02685546875$ memory_percent‚97.02913761138916 cpu‚25.698589999859497 disk.md0.out‚ 692.50390625 disk.dm-0.in‚ 539.2421875# proc.memory.rssMB‚ 3023.18359375* proc.memory.percent‚18.452048301696777 proc.cpu.threads‚17 network.recv‚ 3.91101e+06) disk./.usagePercent‚76.20983710799149% disk./.usageGB‚1170.0115242004395 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚100203.18359375( proc.memory.availableMB‚ 486.74609375 network.sent‚ 2.6709727e+07& disk.nvme1n1.out‚47964.02685546875$ disk.dm-0.out‚131139.23779296875" disk.dm-1.out‚15580.9716796875 disk.rbd0.out‚0–ì˜*œ¡4‚(:‘ žÉöÒ”ûË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚279.838' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.27933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k.h؆¢4‚(:û ­ÉöÒ’—ÆË$ disk.dm-0.out‚131455.73193359375" disk.dm-1.out‚15581.0615234375 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375( proc.memory.availableMB‚ 486.73828125 cpu‚25.698838883730566 network.sent‚ 2.6715718e+07) disk./.usagePercent‚76.20565081784127 disk.md0.out‚ 692.86328125 disk.rbd0.in‚0$ memory_percent‚97.02918529510498( disk.nvme0c0n1.out‚100446.1357421875" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚48038.44189453125* proc.memory.percent‚18.452048301696777 proc.cpu.threads‚17 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0n1.out‚100446.1357421875( disk.nvme1c1n1.out‚48038.44189453125# proc.memory.rssMB‚ 3023.18359375 network.recv‚ 3.91454e+06% disk./.usageGB‚1169.9472541809082¡à3ž£4‚(:“ ­ÉöÒØù¥Ì gpu.0.gpu‚97.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.172' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.05733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!ŠÒ4¤4/ _wandbruntime‚15081  _runtime‚15081œÓ¥Ê¥4Ä _step‚1369 & lr-AdamW/pg1‚4.326380606937963e-05 & lr-AdamW/pg2‚4.326380606937963e-05 trainer/global_step‚16649 % _timestamp‚1.7845219084081805e+09 _runtime‚15081.983897125NÂýØϦ4É trainer/global_step‚16649 % _timestamp‚1.7845219084081805e+09 _runtime‚15081.983897125 _step‚1369 & lr-AdamW/pg1‚4.326380606937963e-05 & lr-AdamW/pg2‚4.326380606937963e-05Ù ð3¸W4§4/ _wandbruntime‚15081  _runtime‚15081F6·O (¨4†P % _timestamp‚1.7845219084094834e+09 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2533527672880560 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19338623288832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39732759 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134792481 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299530300 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19309107280126 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2552837120721960 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164727674 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚572952254432768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚1370 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2548442134644224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚15081.985165727 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2548442134644224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68778228 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚16649 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2567780416508928 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29045469 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164727674 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19446802469888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2533527655841200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29045472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19446916127744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 299520155 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚572952386702848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164737817 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164737817 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19338623288832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553505451962880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553505470575104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2548442151683584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134792481 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134792483 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19309447841400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2548442151683584 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19338281864704 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184€‚÷Ü©4‹P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134792481 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134792483 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19309447841400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2548442151683584 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19338281864704 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 % _timestamp‚1.7845219084094834e+09 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2533527672880560 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19338623288832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39732759 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MDDeviceStatsMonîþ6Ô5#itor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134792481 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299530300 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19309107280126 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2552837120721960 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164727674 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚572952254432768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚1370 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2548442134644224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚15081.985165727 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2548442134644224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68778228 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚16649 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2567780416508928 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29045469 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164727674 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19446802469888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2533527655841200 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29045472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19446916127744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 299520155 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚572952386702848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164737817 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164737817 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19338623288832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553505451962880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553505470575104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2548442151683584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856Ú )RÊ4ª4/ _wandbruntime‚15082  _runtime‚15082AHŸÑ'«4N =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299547763 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19339733394944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164737330 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2533675086508672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2567930159580672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2548590409146368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299547763 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29047206 _step‚1371 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29047203 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2548590426185728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299537604 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚553538650845184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19310556229764 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68782211 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚572986680242176 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299537604 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2548590409146368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134800276 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19447917843456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19310215668378 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164737330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚572986550076416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68782504 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2567930159580672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39735298 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2548590426185728 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2552985659777796 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 % _timestamp‚1.7845219090419571e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19339391963648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚16649 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2552985302177050 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2567929801110016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19339391963648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553538632232960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3ÝŠK‰'¬4•N FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68782211 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚572986680242176 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299537604 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2548590409146368 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134800276 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19447917843456 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19310215668378 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164737330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚572986550076416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68782504 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2567930159580672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39735298 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2548590426185728 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2552985659777796 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 % _timestamp‚1.7845219090419571e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19339391963648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚16649 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2552985302177050 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2567929801110016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19339391963648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553538632232960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299547763 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19339733394944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164737330 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2533675086508672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2567930159580672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2548590409146368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299547763 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29047206 _step‚1371 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29047203 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2548590426185728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299537604 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚553538650845184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19310556229764 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184Û ‚á4­4/ _wandbruntime‚15082  _runtime‚15082¢Š.5®4‹ 0loss/FBt1/detection_loss‚0.32124834050318896 0loss/FBt2/detection_loss‚0.26254651939530893 %loss/total_loss‚6.700172117954887 % _timestamp‚1.7845219090431209e+09 _step‚1372 /loss/FRt1/detection_loss‚0.5115727029757479 /loss/FRt2/detection_loss‚1.4400292755339053 trainer/global_step‚16649 3loss/SRqqt1/assignment_loss‚0.39350290582849246 1loss/SRqqt2/assignment_loss‚0.465579982973256 3loss/FBt1/assignment_loss‚0.0036260132553052604 3loss/FBt2/assignment_loss‚4.251423790106683e-05 0loss/FRt1/assignment_loss‚1.4787997262136594 0loss/FRt2/assignment_loss‚0.6286180344369184 1loss/SRqqt2/detection_loss‚0.6052870634853306 _runtime‚15082.618441035 epoch‚41 1loss/SRqqt1/detection_loss‚0.5893190391158732b8~ܯ4 0loss/FBt1/detection_loss‚0.32124834050318896 0loss/FBt2/detection_loss‚0.26254651939530893 %loss/total_loss‚6.700172117954887 % _timestamp‚1.7845219090431209e+09 _step‚1372 /loss/FRt1/detection_loss‚0.5115727029757479 /loss/FRt2/detection_loss‚1.4400292755339053 trainer/global_step‚16649 3loss/SRqqt1/assignment_loss‚0.39350290582849246 1loss/SRqqt2/assignment_loss‚0.465579982973256 3loss/FBt1/assignment_loss‚0.0036260132553052604 3loss/FBt2/assignment_loss‚4.251423790106683e-05 0loss/FRt1/assignment_loss‚1.4787997262136594 0loss/FRt2/assignment_loss‚0.6286180344369184 1loss/SRqqt2/detection_loss‚0.6052870634853306 _runtime‚15082.618441035 epoch‚41 1loss/SRqqt1/detection_loss‚0.5893190391158732Ü Z›Pž°4‚(:“ ¼ÉöÒ¥Î¾Ë gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.567' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.52233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôÈk±4‚(:ö ¼ÉöÒà®òË% disk./.usageGB‚1169.9475784301758 disk.rbd0.in‚0( disk.nvme0c0n1.out‚100447.2158203125 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚48038.99365234375$ disk.dm-0.out‚131456.69677734375! disk.dm-1.out‚15581.185546875$ memory_percent‚97.03102111816406 network.sent‚ 2.673341e+07" disk.nvme0c0n1.in‚ 309.37109375 disk.md0.in‚0# proc.memory.rssMB‚ 3023.18359375 cpu‚25.699136697910017 network.recv‚ 3.918219e+06 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚100447.2158203125" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚48038.99365234375 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0$ proc.memory.availableMB‚486.4375) disk./.usagePercent‚76.20567193813228 disk.rbd0.out‚0 disk.md0.out‚ 693.11328125* proc.memory.percent‚18.452048301696777 proc.cpu.threads‚17øîk¦²4‚(:ó ËÉöÒŽ¨ÕË cpu‚25.699435950762265 proc.cpu.threads‚17 network.recv‚ 3.923385e+06# disk.nvme0n1.out‚100452.8828125& disk.nvme1n1.out‚48039.99072265625 disk.rbd0.in‚0% disk.nvme0c0n1.out‚100452.8828125# memory_percent‚97.0294713973999# proc.memory.rssMB‚ 3023.18359375* proc.memory.percent‚18.452048301696777% disk./.usageGB‚1169.9481086730957 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.rbd0.out‚0 disk.dm-0.in‚ 539.2421875 network.sent‚ 2.6783914e+07) disk./.usagePercent‚76.20570647601991" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚48039.99072265625 ÃwŒØdisk.md0.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚15581.236328125 disk.md0.out‚ 693.4765625$ disk.dm-0.out‚131462.54248046875" disk.nvme0c0n1.in‚ 309.37109375( proc.memory.availableMB‚ 486.69140625úMMÀž³4‚(:“ ËÉöÒÒª½Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.619' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.53966666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€Ì±³‡´4‚(:ü ÚÉö҈˺Ë) disk./.usagePercent‚76.20570374280578 disk.md0.in‚0" disk.dm-1.out‚15581.3291015625 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375( proc.memory.availableMB‚ 486.66015625* proc.memory.percent‚18.452048301696777 network.sent‚ 2.6788754e+07 network.recv‚ 3.926925e+06% disk./.usageGB‚1169.9480667114258 disk.rbd0.out‚0( disk.nvme1c1n1.out‚48040.51416015625$ disk.dm-0.out‚131463.55029296875" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚100454.0712890625 disk.nvme0n1.in‚ 309.37109375& disk.nvme1n1.out‚48040.51416015625# proc.memory.rssMB‚ 3023.18359375 proc.cpu.threads‚17 disk.md0.out‚ 693.75390625 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0& disk.nvme0n1.out‚100454.0712890625$ memory_percent‚97.02966213226318 cpu‚25.699206020088457åÕëížµ4‚(:“ ÚÉöÒú¯þË gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚271.978' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.65933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚æa`4¶4/ _wandbruntime‚15127  _runtime‚15127û‘HÊ·4Ä trainer/global_step‚16699 % _timestamp‚1.7845219544037182e+09 _runtime‚15127.979364053 _step‚1373 & lr-AdamW/pg1‚4.201241466933401e-05 & lr-AdamW/pg2‚4.201241466933401e-05&2ƒÏ¸4É % _timestamp‚1.7845219544037182e+09 _runtime‚15127.979364053 _step‚1373 & lr-AdamW/pg1‚4.201241466933401e-05 & lr-AdamW/pg2‚4.201241466933401e-05 trainer/global_step‚16699Ý eµœ4¹4/ _wandbruntime‚15127  _runtime‚15127Á1Þ• (º4…P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19393850966528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135182133 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2575249710717952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 trainer/global_step‚16699 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2575249710717952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2555855859751424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19364931054808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135182131 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19364590493534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165222469 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚574667123969536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2560264137308968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165222469 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135182133 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2555855876790784 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135182131 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39847009 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29132169 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚555164465466880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2575250069181440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68979461 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19394192390656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2555855876790784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92686336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165212326 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2575250069181440 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0JÃŒ…(»4ŠP P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165222469 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚574667123969536 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2560264137308968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165222469 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135182133 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2555855876790784 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135182131 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39847009 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29132169 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚555164465466880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2575250069181440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68979461 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19394192390656 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2555855876790784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92686336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165212326 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2575250069181440 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19393850966528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135182133 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2575249710717952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 trainer/global_step‚16699 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2575249710717952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2555855859751424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19364931054808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135182131 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19364590493534 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0Þ ‡…4¼4/ _wandbruntime‚15128  _runtime‚15128WYFÁ%½4N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845219550355554e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29133906 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39849281 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68983184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚555197645736960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚555197664349184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚574701536344064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚16699 _step‚1375 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2556004151292928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135189924 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19503871994880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29133903 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19395302496768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19503770927104 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19365698881786 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2560412676364804 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2541046636921632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19366039443172 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300411906 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2541046619882272 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚15128.611356808 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165221982 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2556004134253568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2560412318764058 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135189926 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135189924 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300411906 PBDeÃg¡UviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19395302496768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300422065 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224îW<ÿ'¾4•N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚574701536344064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚16699 _step‚1375 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2556004151292928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135189924 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19503871994880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29133903 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19395302496768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19503770927104 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19365698881786 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2560412676364804 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2541046636921632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19366039443172 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300411906 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2541046619882272 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚15128.611356808 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165221982 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2556004134253568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2560412318764058 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135189926 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135189924 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300411906 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19395302496768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300422065 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845219550355554e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29133906 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39849281 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68983184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚555197645736960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚555197664349184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864ß s·‘4¿4/ _wandbruntime‚15128  _runtime‚15128xM…gÀ4‹ trainer/global_step‚16699 /loss/FBt2/detection_loss‚0.4015330007943489 _step‚1376 epoch‚41 4loss/FBt1/assignment_loss‚0.00033255357191576494 3loss/FBt2/assignment_loss‚0.0014777617955095265 /loss/FRt1/detection_loss‚0.5001852087047756 1loss/SRqqt2/detection_loss‚0.6783443822117522 _runtime‚15128.612009619 0loss/FRt2/assignment_loss‚0.6286539621924422 1loss/SRqqt1/detection_loss‚0.6531101748684011 /loss/FBt1/detection_loss‚0.4253380088288449 %loss/total_loss‚7.182002742665032 $ _timestamp‚1.784521955036711e+09 0loss/FRt1/assignment_loss‚1.5440828965864755 3loss/SRqqt1/assignment_loss‚0.44740953107757575 3loss/SRqqt2/assignment_loss‚0.44615611347032647 /loss/FRt2/detection_loss‚1.4553791485626641ºDó?Á4 %loss/total_loss‚7.182002742665032 $ _timestamp‚1.784521955036711e+09 0loss/FRt1/assignment_loss‚1.5440828965864755 3loss/SRqqt1/assignment_loss‚0.44740953107757575 3loss/SRqqt2/assignment_loss‚0.44615611347032647 /loss/FRt2/detection_loss‚1.4553791485626641 trainer/global_step‚16699 /loss/FBt2/detection_loss‚0.4015330007943489 _step‚1376 epoch‚41 4loss/FBt1/assignment_loss‚0.00033255357191576494 3loss/FBt2/assignment_loss‚0.0014777617955095265 /loss/FRt1/detection_loss‚0.5001852087047756 1loss/SRqqt2/detection_loss‚0.6783443822117522 _runtime‚15128.612009619 0loss/FRt2/assignment_loss‚0.6286539621924422 1loss/SRqqt1/detection_loss‚0.6531101748684011 /loss/FBt1/detection_loss‚0.4253380088288449à ®­}Â4‚(:ò éÉöÒæ³£Ë network.sent‚ 2.6791718e+07' disk./.usagePercent‚76.205708463812% disk./.usageGB‚1169.9481391906738$ disk.nvme1n1.out‚48042.521484375# memory_percent‚97.0313549041748' proc.memory.availableMB‚ 486.3828125* proc.memory.percent‚18.452048301696777 cpu‚25.698959948151224& disk.nvme0n1.out‚100463.1396484375" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 309.37109375& disk.nvme1c1n1.out‚48042.521484375" disk.dm-1.out‚15581.4072265625 disk.rbd0.out‚0# proc.memory.rssMB‚ 3023.18359375 network.recv‚ 3.930181e+06 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 694.04296875" disk.dm-0.out‚131473.908203125 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚100463.1396484375mCR“Ã4‚(:ˆ éÉöÒÝ‹™Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚138.144' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚46.048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñj{Ä4‚(:ð øÉöÒ߬Ë* proc.memory.percent‚18.452048301696777 network.sent‚ 2.6855428e+07 network.recv‚ 3.935594e+06% disk./.usageGB‚1170.0026969909668" disk.dm-0.out‚131490.056640625 disk.rbd0.out‚0 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 694.25390625# proc.memory.rssMB‚ 3023.18359375 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375 disk.md0.in‚0( proc.memory.availableMB‚ 486.62890625 proc.cpu.threads‚17) disk./.usagePercent‚76.20926213912827# disk.dm-1.out‚15581.50048828125! disk.nvme1n1.out‚ 48042.953125 cpu‚25.698729941434 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0) disk.nvme0c0n1.out‚100479.41552734375' disk.nvme0n1.out‚100479.41552734375" disk.nvme1c1n1.in‚ 229.87109375# disk.nvme1c1n1.out‚ 48042.953125$ memory_percent‚97.02985286712646⿸,žÅ4‚(:“ øÉöÒ³úˆÌ gpu.0.gpu‚90.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚285.204' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.06800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î„V„Æ4‚(:ù ‡ÊöÒª¢¬Ë% disk./.usageGB‚1170.1852722167969 disk.dm-0.in‚ 539.2421875# disk.dm-0.out‚131720.6005859375 disk.rbd0.in‚0' disk.nvme0n1.out‚100640.85498046875' disk.nvme1c1n1.out‚48113.3056640625( proc.memory.availableMB‚ 486.62890625 proc.cpu.threads‚17 network.sent‚ 2.6860296e+07) disk./.usagePercent‚76.22115435380536# disk.dm-1.out‚15581.63720703125) disk.nvme0c0n1.out‚100640.85498046875" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375* proc.memory.percent‚18.452048301696777 cpu‚25.698484195921793 network.recv‚ 3.939011e+06 disk.md0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375% disk.nvme1n1.out‚48113.3056640625$ memory_percent‚97.02985286712646# proc.memory.rssMB‚ 3023.18359375 disk.md0.out‚ 694.78125 disk.dm-1.in‚0 disk.rbd0.out‚0éàŽ”žÇ4‚(:“ ‡ÊöÒ†Œ‚Ì gpu.0.gpu‚91.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.825' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.60833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00€4È4/ _wandbruntime‚15173  _runtime‚151736O,ÊÉ4Ä _step‚1377 & lr-AdamW/pg1‚4.077805864405069e-05 & lr-AdamW/pg2‚4.077805864405069e-05 trainer/global_step‚16749 % _timestamp‚1.7845219998601193e+09 _runtime‚15173.435739085ntÀÏÊ4É & lr-AdamW/pg1‚4.077805864405069e-05 & lr-AdamW/pg2‚4.077805864405069e-05 trainer/global_step‚16749 % _timestamp‚1.7845219998601193e+09 _runtime‚15173.435739085 _step‚1377á Ä’¢™4Ë4/ _wandbruntime‚15173  _runtime‚15173Ò wt (Ì4†P JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19449420068352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165707121 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19449420068352 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135571781 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19449761492480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2582719004926976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2563269601897984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19558489770496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 301268759 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135571783 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69181056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165696978 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚16749 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2548270722588400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845219998612478e+09 _step‚1378 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135571781 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19558597136896 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2582719363390464 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135571783 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2563269584858624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚576382094720000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚15173.436873097 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165707121 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19449761492480 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2563269584858624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39961895 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚556823497583104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29218872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2548270739627760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29218869 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19420073706942 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2582719363390464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2563269601897984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2567691153895976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165696978 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2582719004926976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69180764 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452^~­tfÍ4‹P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135571783 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2563269584858624 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚576382094720000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚15173.436873097 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165707121 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19449761492480 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2563269584858624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39961895 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚556823497583104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29218872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2548270739627760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29218869 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19420073706942 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2582719363390464 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2563269601897984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2567691153895976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165696978 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2582719004926976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69180764 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19449420068352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165707121 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19449420068352 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135571781 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19449761492480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2582719004926976 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2563269601897984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 N@DeviceStatsMonitor.on_train_batch_s¤ K«tart/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19558489770496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 301268759 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135571783 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69181056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165696978 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 trainer/global_step‚16749 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2548270722588400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845219998612478e+09 _step‚1378 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135571781 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19558597136896 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2582719363390464â b°:4Î4/ _wandbruntime‚15174  _runtime‚15174üRúü'Ï4N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39964122 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19559602440704 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2582868747998720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69184725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39964414 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚556856677853184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2563417859360768 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 $ _timestamp‚1.784522000495663e+09 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19450530167296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 trainer/global_step‚16749 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165716791 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚576416391847424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576416261681664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 301296367 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2567839335351066 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2563417876400128 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2582868389528064 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2548418170295232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚1379 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135579576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19559713994240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2548418153255872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19421522656580 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19450530167296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135579576 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19450530167296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 trainer/global_step‚16749 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165716791 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚576416391847424 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576416261681664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 301296367 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2567839335351066 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2563417876400128 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2582868389528064 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2548418170295232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚1379 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135579576 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19559713994240 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2548418153255872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19421522656580 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19450530167296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135579576 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2582868747998720 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69184725 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39964414 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚556856677853184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2563417859360768 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 $ _timestamp‚1.784522000495663e+09ã ~â›Ó4Ñ4/ _wandbruntime‚15174  _runtime‚151744ìÝûÒ4‹ trainer/global_step‚16749 $ _timestamp‚1.784522000496806e+09 2loss/SRqqt1/assignment_loss‚0.4443223450420444 /loss/FRt2/detection_loss‚1.3973420154624714 0loss/FRt2/assignment_loss‚0.5678412931926701 /loss/FRt1/detection_loss‚0.4514019908357362 0loss/FBt1/detection_loss‚0.26950506521035406 _step‚1380 2loss/SRqqt2/assignment_loss‚0.4095347025129669 3loss/FBt2/assignment_loss‚3.138529631451513e-05 1loss/SRqqt1/detection_loss‚0.5801435860332956 1loss/SRqqt2/detection_loss‚0.6261106112631109 &loss/total_loss‚6.5505409771372936 epoch‚41 _runtime‚15174.072111792 0loss/FRt1/assignment_loss‚1.4601171530141366 3loss/FBt1/assignment_loss‚0.0021908363654083696 0loss/FBt2/detection_loss‚0.34199999290878486AOjÎÓ4 0loss/FBt2/detection_loss‚0.34199999290878486 trainer/global_step‚16749 $ _timestamp‚1.784522000496806e+09 2loss/SRqqt1/assignment_loss‚0.4443223450420444 /loss/FRt2/detection_loss‚1.3973420154624714 0loss/FRt2/assignment_loss‚0.5678412931926701 /loss/FRt1/detection_loss‚0.4514019908357362 0loss/FBt1/detection_loss‚0.26950506521035406 _step‚1380 2loss/SRqqt2/assignment_loss‚0.4095347025129669 3loss/FBt2/assignment_loss‚3.138529631451513e-05 1loss/SRqqt1/detection_loss‚0.5801435860332956 1loss/SRqqt2/detection_loss‚0.6261106112631109 &loss/total_loss‚6.5505409771372936 epoch‚41 _runtime‚15174.072111792 0loss/FRt1/assignment_loss‚1.4601171530141366 3loss/FBt1/assignment_loss‚0.0021908363654083696ä þÒ¢(†Ô4‚(:û –ÊöÒ‹§îË( disk.nvme1c1n1.out‚48113.81787109375 network.recv‚ 3.942349e+06 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0n1.out‚100644.11865234375 disk.dm-0.in‚ 539.2421875( proc.memory.availableMB‚ 486.58984375# proc.memory.rssMB‚ 3023.18359375) disk./.usagePercent‚76.22117298935625! disk.dm-1.out‚15581.818359375& disk.nvme1n1.out‚48113.81787109375 cpu‚25.698253535805794 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375) disk.nvme0c0n1.out‚100644.11865234375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0" disk.dm-0.out‚131723.728515625* proc.memory.percent‚18.452048301696777 network.sent‚ 2.6877984e+07% disk./.usageGB‚1170.1855583190918 disk.nvme0n1.in‚ 309.37109375 disk.md0.out‚ 694.98828125$ memory_percent‚97.03009128570557 proc.cpu.threads‚17Î2øÕ4‚(:’ –ÊöÒ㬵Ì gpu.0.gpu‚36.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚247.12' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚82.37333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04,à ƒÖ4‚(:ø ¥ÊöÒý°ÌË disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚97.03161716461182 cpu‚25.697993253775252" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚48116.38525390625 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875 disk.dm-0.out‚ 131733.578125! disk.dm-1.out‚15581.947265625( proc.memory.availableMB‚ 486.33984375 network.sent‚ 2.6928489e+07 network.recv‚ 3.947168e+06) disk./.usagePercent‚76.22116826835003) disk.nvme0c0n1.out‚100652.44580078125 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚100652.44580078125 disk.nvme1n1.in‚ 229.87109375* proc.memory.percent‚18.452048301696777% disk./.usageGB‚1170.1854858398438# proc.memory.rssMB‚ 3023.18359375 proc.cpu.threads‚17& disk.nvme1n1.out‚48116.38525390625 disk.md0.out‚ 695.41015625" disk.nvme0c0n1.in‚ 309.37109375 disk.dm-1.in‚0×ßÉ“×4‚(:ˆ ¥ÊöÒš˜Ì gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.218' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«è:zØ4‚(:ï ´ÊöÒÆšžË% disk./.usageGB‚1170.1853218078613 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme1n1.out‚ 48117.203125# proc.memory.rssMB‚ 3023.18359375 proc.cpu.threads‚17) disk./.usagePercent‚76.22115758396751# disk.dm-1.out‚15582.05322265625 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚100654.36962890625" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 695.9609375# disk.dm-0.out‚131735.0322265625$ memory_percent‚97.03311920166016) disk.nvme0c0n1.out‚100654.36962890625 disk.md0.in‚0% proc.memory.availableMB‚ 486.09375 cpu‚25.69775020382675 network.sent‚ 2.6931512e+07 network.recv‚ 3.951041e+06" disk.nvme0c0n1.in‚ 309.37109375# disk.nvme1c1n1.out‚ 48117.203125 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875* proc.memory.percent‚18.452048301696777ÛÐý±“Ù4‚(:ˆ ´ÊöÒâöúË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.215' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûûƒ74Ú4/ _wandbruntime‚15219  _runtime‚15219,î 1ÉÛ4à _step‚1381 & lr-AdamW/pg1‚3.956081727391077e-05 & lr-AdamW/pg2‚3.956081727391077e-05 trainer/global_step‚16799 $ _timestamp‚1.784522045907125e+09 _runtime‚15219.482741957•êóüÎÜ4È $ _timestamp‚1.784522045907125e+09 _runtime‚15219.482741957 _step‚1381 & lr-AdamW/pg1‚3.956081727391077e-05 & lr-AdamW/pg2‚3.956081727391077e-05 trainer/global_step‚16799å á+¨,4Ý4/ _wandbruntime‚15219  _runtime‚15219rÔq§ÔÞ4†P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135961431 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2570683327005184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40075733 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29305572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2590188299136000 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2570683309965824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29305569 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19505330594304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69381591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135961433 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚578096949493760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚16799 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69381302 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2575118170482984 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2555642255962000 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845220459082212e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166191773 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19504989170176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302153206 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166191773 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2555642273001360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2570683309965824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19504989170176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚558482511087104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2590188299136000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40076019 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19614438406656 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2575117812882350 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚15219.483841958 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚578096823515136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚1382 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2570683327005184 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19614331040256 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19475556920350 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488²Ó›(ß4‹P N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2590188299136000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40076019 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19614438406656 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2575117812882350 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚15219.483841958 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚578096823515136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _step‚1382 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2570683327005184 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19614331040256 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19475556920350 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135961431 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2570683327005184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40075733 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29305572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2590188299136000 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2570683309965824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29305569 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19505330594304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69381591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135961433 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚578096949493760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚16799 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69381302 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2575118170482984 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2555642255962000 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845220459082212e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166191773 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19504989170176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302153206 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166191773 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2555642273001360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2570683309965824 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19504989170176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚558482511087104æ £Wpp4à4/ _wandbruntime‚15220  _runtime‚15220šÎe'á4N OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2575266351938074 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40078012 _runtime‚15220.116657579 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29307303 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19506099269120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚578131243343872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19615450918912 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166201443 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302160510 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2570831601507328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2555789703668832 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166191286 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2555789686629472 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2590338042207744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19477005869988 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚558515672744960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135969226 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166191286 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845220465408533e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2590338042207744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2570831584467968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19506440700416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40078290 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19476665308602 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135969226 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29307306 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚578131123663872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚16799 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2590337683737088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302170669 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19615551986688 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69385315 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302160510 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135969224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135969224 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135969226 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166191286 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845220465408533e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2590338042207744 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2570831584467968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19506440700416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40078290 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19476665308602 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135969226 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29307306 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚578131123663872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚16799 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2590337683737088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302170669 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytGÍrzÑes.small_pool.allocated‚19615551986688 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69385315 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302160510 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135969224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135969224 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19506099269120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚578131243343872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19615450918912 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166201443 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302160510 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2570831601507328 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ç WEBõ4ã4/ _wandbruntime‚15220  _runtime‚152208ˆ¶ðä4Š _runtime‚15220.11734144 2loss/SRqqt2/assignment_loss‚0.4315194966113564 0loss/FRt1/detection_loss‚0.46891431616769363 1loss/SRqqt2/detection_loss‚0.5819389644370685 0loss/FBt1/detection_loss‚0.29927347468949833 0loss/FBt2/detection_loss‚0.32851484185717394 trainer/global_step‚16799 0loss/FRt1/assignment_loss‚1.5399754089262427 0loss/FRt2/assignment_loss‚0.7012320969378549 2loss/FBt2/assignment_loss‚0.007159288187284048 1loss/SRqqt1/detection_loss‚0.5888236807425238 _step‚1384 3loss/FBt1/assignment_loss‚8.795439329236091e-05 /loss/FRt2/detection_loss‚1.4765700185437505 epoch‚41 2loss/SRqqt1/assignment_loss‚0.4466507446914059 %loss/total_loss‚6.870660286185146 % _timestamp‚1.7845220465420163e+09~%?å4 3loss/FBt1/assignment_loss‚8.795439329236091e-05 /loss/FRt2/detection_loss‚1.4765700185437505 epoch‚41 2loss/SRqqt1/assignment_loss‚0.4466507446914059 %loss/total_loss‚6.870660286185146 % _timestamp‚1.7845220465420163e+09 _runtime‚15220.11734144 2loss/SRqqt2/assignment_loss‚0.4315194966113564 0loss/FRt1/detection_loss‚0.46891431616769363 1loss/SRqqt2/detection_loss‚0.5819389644370685 0loss/FBt1/detection_loss‚0.29927347468949833 0loss/FBt2/detection_loss‚0.32851484185717394 trainer/global_step‚16799 0loss/FRt1/assignment_loss‚1.5399754089262427 0loss/FRt2/assignment_loss‚0.7012320969378549 2loss/FBt2/assignment_loss‚0.007159288187284048 1loss/SRqqt1/detection_loss‚0.5888236807425238 _step‚1384è C>ž¥æ4‚(:ô ÃÊöÒë­îË disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 2.6949425e+07 network.recv‚ 3.953769e+06 disk.nvme0n1.in‚ 309.37109375% disk.nvme0n1.out‚100657.904296875' disk.nvme1c1n1.out‚48118.3583984375$ disk.dm-0.out‚131738.68896484375# disk.dm-1.out‚15582.19775390625$ memory_percent‚97.03812599182129" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0c0n1.out‚100657.904296875' proc.memory.availableMB‚ 485.2734375# proc.memory.rssMB‚ 3024.45703125) disk./.usagePercent‚76.22115261448728 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 696.3828125) proc.memory.percent‚18.45982074737549 cpu‚25.69822588215814$ disk./.usageGB‚1170.185245513916% disk.nvme1n1.out‚48118.3583984375 disk.dm-0.in‚ 539.2421875Ž%žç4‚(:“ ÃÊöÒ“œ³Ì gpu.0.gpu‚97.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.425' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.47500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’‚n4è4/ _wandbruntime‚15237  _runtime‚15237þ2=é4”z +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6001429557800293 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6071388721466064 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.614380419254303 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4446840286254883 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3390961289405823 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6912720799446106 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9098871350288391 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5876421928405762 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6599446535110474 +jet/accuracy_3_of_4‚0.08844424784183502 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 +particle/sensitivity‚0.6426360607147217 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7369792461395264 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29327842593193054 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 _step‚1385 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6421918272972107 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6041666269302368 *jet/accuracy_2_of_3‚0.5531607270240784 /particle/accuracy_4_of_4‚0.9545081257820129 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6065689325332642 7"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.69861900806427 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9922193884849548 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04452304169535637 particle/accuracy_3_of_4‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8437452912330627 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2800297439098358 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8633031249046326 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6247607469558716 *jet/accuracy_1_of_4‚0.9590424299240112 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5811772346496582 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5609009861946106 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7174937129020691 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08193506300449371 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08747639507055283 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07580661028623581 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08651058375835419 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2918888032436371 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6305180788040161 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.604208767414093 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7350054383277893 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.32225120067596436 particle/accuracy_1_of_5‚1 trainer/global_step‚16799 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5813831090927124 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.869701087474823 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9918209314346313 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5497370958328247 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5609009861946106 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 % _timestamp‚1.7845220636206691e+09 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861111044883728 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5780574083328247 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2218165397644043 particle/accuracy_2_of_5‚1 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.99639493227005 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08132567256689072 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22137844562530518 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04925199970602989 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08644860237836838 *jet/accuracy_1_of_2‚0.7599189877510071 particle/accuracy_2_of_3‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.997460126876831 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8858659863471985 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7106639742851257 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33728185296058655 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.28090548515319824 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7666636109352112 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6169363856315613 (particle/accuracy‚0.8518105745315552 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6523065567016602 7validation_average_jet_accuracy‚0.36368995904922485 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6096037030220032 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23440109193325043 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_4‚1 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4955299496650696 'particle/f_score‚0.6506642699241638 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5865398049354553 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21133092045783997 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_2_of_4‚1 jet/accuracy_4_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9972140789031982 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5035807490348816 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6667784452438354 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.581630527973175 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +particle/specificity‚0.9090238213539124 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8720710277557373 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45426297187805176 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19895687699317932 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8478080034255981 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6290851831436157 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7036837935447693 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2933633327484131 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.23083333671092987 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2918788492679596 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2426290214061737 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.45796629786491394 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2587478458881378 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2800297439098358 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6930528283119202 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2696823477745056 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29470351338386536 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06885191053152084 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33003565669059753 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6671239137649536 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.673681378364563 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5631216168403625 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5052083134651184 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2716773450374603 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2027071714401245 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 particle/accuracy_2_of_2‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2516670823097229 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6877565979957581 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9481417536735535 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5785557627677917 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7666636109352112 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3339785039424896 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6639944314956665 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4984866678714752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 _runtime‚15237.196735025 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7218956351280212 jet/accuracy_1_of_5‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10917679220438004 /particle/accuracy_3_of_3‚0.9994120597839355 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988906383514404 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 epoch‚41 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 particle/accuracy_1_of_1‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +jet/accuracy_2_of_2‚0.15666252374649048 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7555280327796936 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29111742973327637 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_3‚0.9125534296035767 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992322325706482 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28116703033447266 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3140442371368408 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03824549913406372 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0561387874186039 particle/accuracy_1_of_2‚1 'loss/val_total_loss‚7.4028715621492 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2580013573169708 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06885191053152084 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23003479838371277 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2698308229446411 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29131942987442017 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6340419054031372 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5601633191108704 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.268273651599884031ÿ{,!ê4™z :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33728185296058655 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.28090548515319824 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7666636109352112 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6169363856315613 (particle/accuracy‚0.8518105745315552 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6523065567016602 7validation_average_jet_accuracy‚0.36368995904922485 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6096037030220032 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23440109193325043 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_4‚1 jet/accuracy_5_of_5‚0 particle/accuracy_5_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4955299496650696 'particle/f_score‚0.6506642699241638 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5865398049354553 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21133092045783997 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.01670061983168125 particle/accuracy_2_of_4‚1 jet/accuracy_4_of_5‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9972140789031982 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5035807490348816 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6667784452438354 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.581630527973175 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +particle/specificity‚0.9090238213539124 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8720710277557373 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45426297187805176 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19895687699317932 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8478080034255981 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6290851831436157 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7036837935447693 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2933633327484131 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.23083333671092987 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2918788492679596 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2426290214061737 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.45796629786491394 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2587478458881378 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2800297439098358 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6930528283119202 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2696823477745056 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29470351338386536 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06885191053152084 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33003565669059753 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6671239137649536 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.673681378364563 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5631216168403625 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5052083134651184 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2716773450374603 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2027071714401245 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 particle/accuracy_2_of_2‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2516670823097229 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6877565979957581 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9481417536735535 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5785557627677917 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7666636109352112 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3339785039424896 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6639944314956665 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4984866678714752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 _runtime‚15237.196735025 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7218956351280212 jet/accuracy_1_of_5‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10917679220438004 /particle/accuracy_3_of_3‚0.9994120597839355 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988906383514404 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 epoch‚41 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 particle/accuracy_1_of_1‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +jet/accuracy_2_of_2‚0.15666252374649048 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7555280327796936 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29111742973327637 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_3‚0.9125534296035767 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992322325706482 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28116703033447266 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3140442371368408 & Purity/2FRt1SRqqt*FBt/FBt_purity‚4w¨ó1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03824549913406372 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0561387874186039 particle/accuracy_1_of_2‚1 'loss/val_total_loss‚7.4028715621492 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2580013573169708 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06885191053152084 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23003479838371277 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2698308229446411 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29131942987442017 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6340419054031372 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5601633191108704 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26827365159988403 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6001429557800293 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6071388721466064 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.614380419254303 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4446840286254883 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3390961289405823 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6912720799446106 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9098871350288391 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5876421928405762 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6599446535110474 +jet/accuracy_3_of_4‚0.08844424784183502 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 +particle/sensitivity‚0.6426360607147217 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7369792461395264 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29327842593193054 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 _step‚1385 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6421918272972107 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6041666269302368 *jet/accuracy_2_of_3‚0.5531607270240784 /particle/accuracy_4_of_4‚0.9545081257820129 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6065689325332642 7"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.69861900806427 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9922193884849548 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04452304169535637 particle/accuracy_3_of_4‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8437452912330627 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2800297439098358 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8633031249046326 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6247607469558716 *jet/accuracy_1_of_4‚0.9590424299240112 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5811772346496582 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5609009861946106 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7174937129020691 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08193506300449371 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08747639507055283 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07580661028623581 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.08651058375835419 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2918888032436371 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6305180788040161 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.604208767414093 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7350054383277893 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.32225120067596436 particle/accuracy_1_of_5‚1 trainer/global_step‚16799 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5813831090927124 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.869701087474823 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9918209314346313 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5497370958328247 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5609009861946106 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 % _timestamp‚1.7845220636206691e+09 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861111044883728 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5780574083328247 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2218165397644043 particle/accuracy_2_of_5‚1 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.99639493227005 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08132567256689072 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22137844562530518 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04925199970602989 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08644860237836838 *jet/accuracy_1_of_2‚0.7599189877510071 particle/accuracy_2_of_3‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.997460126876831 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8858659863471985 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7106639742851257 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0é š˜ë4‚(:ö ÒÊöÒ”‡Ì disk.md0.in‚0" proc.memory.rssMB‚ 3025.7734375 network.sent‚ 2.7010796e+07 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚131739.26708984375" disk.dm-1.out‚15582.3779296875 disk.rbd0.in‚0$ memory_percent‚97.04535007476807( proc.memory.availableMB‚ 484.08984375) disk.nvme0c0n1.out‚100658.64111328125$ disk.nvme1n1.out‚48118.876953125 disk.dm-1.in‚0 disk.rbd0.out‚0) proc.memory.percent‚18.46785545349121 cpu‚25.700226134400257) disk./.usagePercent‚76.22116628055792% disk./.usageGB‚1170.1854553222656 disk.nvme0n1.in‚ 309.37109375 disk.md0.out‚ 696.609375 proc.cpu.threads‚17 network.recv‚ 3.958937e+06" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0n1.out‚100658.64111328125" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1c1n1.out‚48118.876953125£¼½žì4‚(:“ ÒÊöÒ±ÏÐÌ gpu.0.gpu‚83.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚256.823' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.60766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ĨŸcží4‚(:“ áÊöÒöæË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚270.446' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.14866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô`}'~î4‚(:ó áÊöÒêïˆÌ disk.md0.out‚ 696.91015625( disk.nvme1c1n1.out‚48119.41552734375 disk.nvme1n1.in‚ 229.87109375) proc.memory.percent‚18.46785545349121 cpu‚25.69996400954161 network.recv‚ 3.963978e+06% disk./.usageGB‚1170.1857147216797$ disk.dm-0.out‚131740.34326171875 disk.dm-1.in‚0" disk.dm-1.out‚15582.4677734375" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0c0n1.out‚100659.931640625$ proc.memory.availableMB‚483.3125 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375% disk.nvme0n1.out‚100659.931640625" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.05009460449219" proc.memory.rssMB‚ 3025.7734375 network.sent‚ 2.7055061e+07 disk.md0.in‚0& disk.nvme1n1.out‚48119.41552734375 proc.cpu.threads‚17) disk./.usagePercent‚76.22118317679073“G••žï4‚(:“ ðÊöÒ僾Ë gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚273.847' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.28233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0††¸‚ð4‚(:÷ ðÊöÒ²ÂÏË disk.dm-1.in‚0 network.sent‚ 2.706182e+07 disk.md0.out‚ 697.18359375$ disk.dm-0.out‚131741.20263671875& disk.nvme0n1.out‚100660.8896484375( proc.memory.availableMB‚ 483.02734375" proc.memory.rssMB‚ 3025.7734375 cpu‚25.699736583537728) disk./.usagePercent‚76.22115261448728$ disk./.usageGB‚1170.185245513916 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375) proc.memory.percent‚18.46785545349121" disk.dm-1.out‚15582.6181640625 disk.rbd0.in‚0 network.recv‚ 3.96762e+06 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚48120.07080078125 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875( disk.nvme0c0n1.out‚100660.8896484375 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚48120.07080078125$ memory_percent‚97.05183506011963 proc.cpu.threads‚17ÂŒÈ4ñ4/ _wandbruntime‚15282  _runtime‚15282Y½;Ìò4Æ _step‚1386 ' lr-AdamW/pg1‚3.8360768740052814e-05 ' lr-AdamW/pg2‚3.8360768740052814e-05 trainer/global_step‚16849 % _timestamp‚1.7845221088955834e+09 _runtime‚15282.471209132+W•Ñó4Ë ' lr-AdamW/pg2‚3.8360768740052814e-05 trainer/global_step‚16849 % _timestamp‚1.7845221088955834e+09 _runtime‚15282.471209132 _step‚1386 ' lr-AdamW/pg1‚3.8360768740052814e-05ê Íó«œ4ô4/ _wandbruntime‚15282  _runtime‚15282Í# (õ4ƒP GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303231810 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2584486746927113 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166757919 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136473891 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2564935062383184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2584486389326479 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136473891 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29396385 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚579854974514176 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 303221665 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19580876107776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19581217531904 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136473889 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚579855094201344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2580039378948608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19690990947840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚560164084641280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _step‚1387 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2599620613519872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 " _timestamp‚1.7845221088967e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69626713 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2599620255056384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166757919 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40230607 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69626995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2599620613519872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19581217531904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19690889872896 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2564935045343824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2580039395987968 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2599620255056384 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166747776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29396388 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19551343982655 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2580039378948608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19580876107776 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19551684543929 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚15282.472282714yÌŠµ(ö4ˆP G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29396385 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚579854974514176 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 303221665 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19580876107776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19581217531904 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136473889 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚579855094201344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2580039378948608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19690990947840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚560164084641280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _step‚1387 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2599620613519872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 " _timestamp‚1.7845221088967e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69626713 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2599620255056384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166757919 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40230607 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69626995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2599620613519872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19581217531904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19690889872896 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2564935045343824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2580039395987968 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2599620255056384 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166747776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29396388 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19551343982655 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2580039378948608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19580876107776 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19551684543929 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 _runtime‚15282.472282714 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303231810 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2584486746927113 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166757919 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136473891 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2564935062383184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2584486389326479 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136473891ë 1RDñ4÷4/ _wandbruntime‚15283  _runtime‚15283Æltáùø4N ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19582327638016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19581986206720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ;5DeviceStatsMon«BV¾%itor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2565082476011296 % _timestamp‚1.7845221095265393e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166767589 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29398122 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚579889268267520 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 303239114 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166757432 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚16849 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚579889392141824 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40232585 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19692003356160 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2580187670490112 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2599769998128128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚15283.10233232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29398119 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2584635285982949 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136481682 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2599769998128128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2584634928382203 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2580187670490112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166767589 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19552792932293 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19692108618240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19552452370907 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 303249273 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2580187653450752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136481682 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2599769639657472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303249273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚560197264911360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69630704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2565082493050656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303239114 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136481684 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136481684 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40232585 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19692003356160 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2580187670490112 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2599769998128128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚15283.10233232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29398119 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2584635285982949 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136481682 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2599769998128128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2584634928382203 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2580187670490112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166767589 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19552792932293 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19692108618240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19552452370907 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 303249273 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2580187653450752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136481682 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2599769639657472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303249273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚560197264911360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69630704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2565082493050656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303239114 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136481684 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136481684 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2587453104055808 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2607089907728896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29483088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19746716517376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2591913405913487 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19607167757337 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167232428 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2572306578717424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19636445209600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚581569936446976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136863539 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2591913763514121 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136863539 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167242571 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69828744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2587453121095168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19636445209600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚561823116757504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚16899 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2607089549265408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚581569814662656 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2587453104055808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167242571 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40345381 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2607089907728896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40345381 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2607089907728896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2587453104055808 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2607089907728896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29483088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19746716517376 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2591913405913487 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19607167757337 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167232428 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2572306578717424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19636445209600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚581569936446976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136863539 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2591913763514121 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136863539 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167242571 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69828744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2587453121095168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19636445209600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚561823116757504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚16899 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2607089549265408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚581569814662656 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2587453104055808 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ï ©ˆ@ü4‰5/ _wandbruntime‚15329  _runtime‚15329V÷'Š5N [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2572454009384896 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2587601395597312 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚15329.03455401 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40347574 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136871332 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2607239292337152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚561856278415360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304113416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40347864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2587601395597312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167252241 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304123575 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2607239292337152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19637555308544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7845221554587522e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19637896739840 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2592061944969211 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304123575 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69832393 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚581604235462656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19747938435072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2592062302569957 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19608276145701 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136871332 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19747826881536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304123575 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69832393 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚581604235462656 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19747938435072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2592062302569957 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19608276145701 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136871332 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19747826881536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40347864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_trï¥BÞóain_batch_end/active.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2587601395597312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167252241 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290ð ô¹g¨4Œ5/ _wandbruntime‚15329  _runtime‚15329=u„Ï5Š 2loss/FBt1/assignment_loss‚0.014814891056896144 /loss/FRt2/detection_loss‚1.3767364733660765 1loss/SRqqt2/detection_loss‚0.6338129314294021 /loss/FBt2/detection_loss‚0.5367154623651537 epoch‚42 0loss/FRt1/assignment_loss‚1.4525046277225973 0loss/FRt2/assignment_loss‚0.7087026398453936 3loss/SRqqt2/assignment_loss‚0.49899549702250157 4loss/FBt2/assignment_loss‚0.00018290770030660042 0loss/SRqqt1/detection_loss‚0.548361978732687 0loss/FBt1/detection_loss‚0.23944237381250624 trainer/global_step‚16899 %loss/total_loss‚6.814862459338274 % _timestamp‚1.7845221554598339e+09 /loss/FRt1/detection_loss‚0.4590130058162108 _runtime‚15329.035096631 _step‚1393 2loss/SRqqt1/assignment_loss‚0.3455796704685421>§8óŽ5 /loss/FRt1/detection_loss‚0.4590130058162108 _runtime‚15329.035096631 _step‚1393 2loss/SRqqt1/assignment_loss‚0.3455796704685421 2loss/FBt1/assignment_loss‚0.014814891056896144 /loss/FRt2/detection_loss‚1.3767364733660765 1loss/SRqqt2/detection_loss‚0.6338129314294021 /loss/FBt2/detection_loss‚0.5367154623651537 epoch‚42 0loss/FRt1/assignment_loss‚1.4525046277225973 0loss/FRt2/assignment_loss‚0.7087026398453936 3loss/SRqqt2/assignment_loss‚0.49899549702250157 4loss/FBt2/assignment_loss‚0.00018290770030660042 0loss/SRqqt1/detection_loss‚0.548361978732687 0loss/FBt1/detection_loss‚0.23944237381250624 trainer/global_step‚16899 %loss/total_loss‚6.814862459338274 % _timestamp‚1.7845221554598339e+09ñ Nç$\{5‚(:ð ¬ËöÒ¨üÌ disk.rbd0.out‚0 network.sent‚ 2.7149754e+07 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875 disk.dm-1.out‚ 15582.9765625 disk.nvme0n1.in‚ 309.37109375$ memory_percent‚97.05255031585693% disk./.usageGB‚1170.1848411560059 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚48122.83349609375 disk.md0.out‚698.6875" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚48122.83349609375 cpu‚25.69875879932771 proc.cpu.threads‚17 network.recv‚ 3.982594e+06) disk./.usagePercent‚76.22112627624203$ disk.dm-0.out‚131746.27001953125 disk.rbd0.in‚0% disk.nvme0n1.out‚100666.775390625" disk.nvme1c1n1.in‚ 229.87109375( proc.memory.availableMB‚ 482.91015625) proc.memory.percent‚18.46785545349121 disk.dm-1.in‚0' disk.nvme0c0n1.out‚100666.775390625" proc.memory.rssMB‚ 3025.7734375^¾ý|“5‚(:ˆ ¬ËöÒ³ÑÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚167.916' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚55.972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jq™Ü€‘5‚(:õ »ËöÒíá¸Ë' disk.nvme0n1.out‚100669.09716796875( disk.nvme1c1n1.out‚48123.75927734375& disk.nvme1n1.out‚48123.75927734375 disk.dm-1.in‚0$ memory_percent‚97.05278873443604) proc.memory.percent‚18.46785545349121 cpu‚25.69850161089339 proc.cpu.threads‚17% disk./.usageGB‚1170.1860694885254 disk.rbd0.in‚0" disk.dm-0.out‚131748.755859375) disk./.usagePercent‚76.22120628487382" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 698.984375( proc.memory.availableMB‚ 482.87109375" proc.memory.rssMB‚ 3025.7734375 network.sent‚ 2.7201297e+07 network.recv‚ 3.987415e+06 disk.rbd0.out‚0) disk.nvme0c0n1.out‚100669.09716796875" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875 disk.dm-1.out‚15583.08984375ƒ.“’5‚(:ˆ »ËöÒº—õË gpu.0.gpu‚69.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚272.868' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚90.956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R­«yž“5‚(:“ ÊËöÒ’“ÊË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.214' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.07133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜÑ}Œ”5‚(:ö ÊËöÒ§äæË cpu‚25.69843678621065 proc.cpu.threads‚17& disk.nvme1n1.out‚48124.05712890625 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875" disk.dm-0.out‚131749.701171875 disk.dm-1.in‚0) disk.nvme0c0n1.out‚100670.10302734375# memory_percent‚97.0543384552002' proc.memory.availableMB‚ 482.6171875 network.sent‚ 2.7206126e+07 network.recv‚ 3.99106e+06% disk./.usageGB‚1170.1867141723633 disk.nvme1n1.in‚ 229.87109375" disk.dm-1.out‚15583.1474609375' disk.nvme0n1.out‚100670.10302734375" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚48124.05712890625) disk./.usagePercent‚76.22124827698181 disk.md0.out‚ 699.10546875 disk.nvme0n1.in‚ 309.37109375" proc.memory.rssMB‚ 3025.7734375 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375) proc.memory.percent‚18.46785545349121Ñ*ò‡4•5/ _wandbruntime‚15373  _runtime‚15373ndéZÌ–5Æ _runtime‚15373.646388845 _step‚1394 ' lr-AdamW/pg1‚3.6012557379467715e-05 ' lr-AdamW/pg2‚3.6012557379467715e-05 trainer/global_step‚16949 % _timestamp‚1.7845222000706592e+09ç(Ñ—5Ë % _timestamp‚1.7845222000706592e+09 _runtime‚15373.646388845 _step‚1394 ' lr-AdamW/pg1‚3.6012557379467715e-05 ' lr-AdamW/pg2‚3.6012557379467715e-05 trainer/global_step‚16949ò Q˜—44˜5/ _wandbruntime‚15373  _runtime‚15373vß‹A (™5†P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19802610598400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2594866846202368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167727223 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2614559201937920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167717080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40460113 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7845222000732117e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚563482130261504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19662310409471 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137253191 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19692355735552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40459827 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2614558843474432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167727223 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚15373.648797989 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19692014311424 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2614558843474432 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2599340780101129 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19662650970745 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304980414 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137253189 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29569785 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70029612 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19802496940544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚583284740859904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2599340422500495 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 304970269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _step‚1395 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167717080 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚563482111649280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137253191 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70029901 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2594866829163008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2614559201937920 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2579678112091024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137253189 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19692014311424 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2579678129130384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272³ë´<(š5‹P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19692355735552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40459827 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2614558843474432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167727223 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚15373.648797989 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19692014311424 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2614558843474432 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2599340780101129 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19662650970745 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304980414 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137253189 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29569785 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70029612 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19802496940544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚583284740859904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2599340422500495 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 304970269 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _step‚1395 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167717080 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚563482111649280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137253191 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70029901 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2594866829163008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2614559201937920 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2579678112091024 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137253189 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19692014311424 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2579678129130384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19802610598400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2594866846202368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167727223 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2614559201937920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167717080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40460113 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7845222000732117e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚563482130261504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19662310409471 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137253191ó ^™&F4›5/ _wandbruntime‚15374  _runtime‚15374É(o œ5N \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2595015120704512 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137260984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2579825542758496 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚16949 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167736893 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137260982 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167736893 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845222007030246e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19803610864128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304987718 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304997877 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2599489319156965 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19693465841664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2599488961556219 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19693124410368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167726736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70033615 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚563515310531584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2579825559797856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19693124410368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚583318902783488 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2595015103665152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ÊŒv 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137260984 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2614708586546176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304987718 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40462371 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19663418797723 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137260982 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29571522 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304997877 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2614708586546176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19693465841664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19663759359109 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29571519 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70033893 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167726736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚583319032949248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚15374.278802856ýÈ'5•N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137260982 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29571522 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 304997877 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2614708586546176 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19693465841664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19663759359109 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29571519 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70033893 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167726736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚583319032949248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚15374.278802856 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2595015120704512 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137260984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2579825542758496 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚16949 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167736893 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137260982 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 167736893 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845222007030246e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19803610864128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304987718 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304997877 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2599489319156965 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19693465841664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2599488961556219 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19693124410368 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167726736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70033615 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚563515310531584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2579825559797856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19693124410368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚583318902783488 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2595015103665152 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137260984 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2614708586546176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304987718 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40462371 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19663418797723 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344ô ¨4ž5/ _wandbruntime‚15374  _runtime‚15374\fü¨Ÿ5‰ _step‚1397 3loss/FBt1/assignment_loss‚8.113324795999688e-06 trainer/global_step‚16949 4loss/FBt2/assignment_loss‚3.4367682075135923e-06 /loss/FBt2/detection_loss‚0.3336707416478158 0loss/FRt2/assignment_loss‚0.6783702962282228 3loss/SRqqt1/assignment_loss‚0.38228245876403855 .loss/FRt2/detection_loss‚1.492869484321237 1loss/SRqqt1/detection_loss‚0.6188905809207687 $loss/total_loss‚6.89224884639874 epoch‚42 _runtime‚15374.279380267 /loss/FRt1/assignment_loss‚1.457422735933143 3loss/SRqqt2/assignment_loss‚0.49031396451118636 0loss/FRt1/detection_loss‚0.49879302418286614 0loss/SRqqt2/detection_loss‚0.607383020020699 /loss/FBt1/detection_loss‚0.3322409897757588 % _timestamp‚1.7845222007041125e+09Åœ½ 5Ž 4loss/FBt2/assignment_loss‚3.4367682075135923e-06 /loss/FBt2/detection_loss‚0.3336707416478158 0loss/FRt2/assignment_loss‚0.6783702962282228 3loss/SRqqt1/assignment_loss‚0.38228245876403855 .loss/FRt2/detection_loss‚1.492869484321237 1loss/SRqqt1/detection_loss‚0.6188905809207687 $loss/total_loss‚6.89224884639874 epoch‚42 _runtime‚15374.279380267 /loss/FRt1/assignment_loss‚1.457422735933143 3loss/SRqqt2/assignment_loss‚0.49031396451118636 0loss/FRt1/detection_loss‚0.49879302418286614 0loss/SRqqt2/detection_loss‚0.607383020020699 /loss/FBt1/detection_loss‚0.3322409897757588 % _timestamp‚1.7845222007041125e+09 _step‚1397 3loss/FBt1/assignment_loss‚8.113324795999688e-06 trainer/global_step‚16949õ $ ÀP~¡5‚(:ó ÙËöÒÓªÒË disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375) proc.memory.percent‚18.46785545349121 proc.cpu.threads‚17) disk./.usagePercent‚76.22122044789249 disk.md0.out‚699.4375 disk.dm-0.in‚ 539.2421875( disk.nvme0c0n1.out‚100672.3193359375& proc.memory.availableMB‚ 482.578125( disk.nvme1c1n1.out‚48124.72314453125 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0" disk.dm-1.out‚15583.1943359375# memory_percent‚97.0545768737793 network.recv‚ 3.994268e+06% disk./.usageGB‚1170.1862869262695& disk.nvme0n1.out‚100672.3193359375& disk.nvme1n1.out‚48124.72314453125$ disk.dm-0.out‚131751.82177734375 disk.rbd0.in‚0" proc.memory.rssMB‚ 3025.7734375 cpu‚25.698455102964033 network.sent‚ 2.7209149e+07 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0¶;Ï)ž¢5‚(:“ ÙËöÒïìÉË gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚271.678' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.55933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÐ\w£5‚(:ì èËöÒŠ½ÂË" proc.memory.rssMB‚ 3025.7734375 proc.cpu.threads‚17) disk./.usagePercent‚76.22122641126877 disk.dm-0.in‚ 539.2421875) proc.memory.percent‚18.46785545349121 network.recv‚ 3.999646e+06$ disk./.usageGB‚1170.186378479004( disk.nvme1c1n1.out‚48125.15087890625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0" disk.nvme0c0n1.in‚ 309.37109375$ disk.nvme0c0n1.out‚ 100673.015625 disk.nvme0n1.in‚ 309.37109375 network.sent‚ 2.7272853e+07" disk.nvme1c1n1.in‚ 229.87109375 disk.rbd0.in‚0" disk.nvme0n1.out‚ 100673.015625$ memory_percent‚97.05440998077393( proc.memory.availableMB‚ 482.60546875 cpu‚25.69840869529802& disk.nvme1n1.out‚48125.15087890625 disk.md0.out‚ 699.609375$ disk.dm-0.out‚131752.41552734375 disk.dm-1.in‚0 disk.dm-1.out‚15583.32421875 disk.rbd0.out‚0}ç~’¤5‚(:‡ èËöÒ è¯Ì gpu.0.gpu‚26.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.312' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ty†ž‘¥5‚(:† ÷ËöÒ»®ÂË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.856' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DJï¦5‚(:ö ÷ËöÒ û§Ì network.sent‚ 2.7278483e+07& disk.nvme0n1.out‚100675.4541015625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 700.0234375 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚131754.53564453125 network.recv‚ 4.003115e+06 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚48125.79345703125 disk.dm-1.out‚ 15583.3828125" proc.memory.rssMB‚ 3025.7734375 cpu‚25.698196394298805) disk./.usagePercent‚76.22122417500266% disk./.usageGB‚1170.1863441467285 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚48125.79345703125 disk.dm-1.in‚0$ memory_percent‚97.05440998077393 proc.cpu.threads‚17 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚100675.4541015625( proc.memory.availableMB‚ 482.60546875) proc.memory.percent‚18.46785545349121ƒ[\€4§5/ _wandbruntime‚15419  _runtime‚15419þú¨•Ê¨5Ä trainer/global_step‚16999 % _timestamp‚1.7845222460160673e+09 _runtime‚15419.591734637 _step‚1398 & lr-AdamW/pg1‚3.486454537396837e-05 & lr-AdamW/pg2‚3.486454537396837e-05cM&Ï©5É & lr-AdamW/pg2‚3.486454537396837e-05 trainer/global_step‚16999 % _timestamp‚1.7845222460160673e+09 _runtime‚15419.591734637 _step‚1398 & lr-AdamW/pg1‚3.486454537396837e-05ö é934ª5/ _wandbruntime‚15419  _runtime‚15419—ä («5†P [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚15419.592934609 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚584999443752960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19858425966080 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2587049662503984 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2622028137683456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19858318599680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19718134184153 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19717793622879 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2622028496146944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2622028496146944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29656488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19747924837376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137642841 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137642839 _step‚1399 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305854716 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2622028137683456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚565141125153280 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2602280554270208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29656485 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚565141143765504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168211875 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19747583413248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70231828 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137642839 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2602280554270208 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚584999443752960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19858425966080 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2587049662503984 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2622028137683456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19858318599680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19718134184153 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19717793622879 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2622028496146944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2622028496146944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29656488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19747924837376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137642841 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137642839 _step‚1399 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305854716 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2622028137683456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚565141125153280 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2602280554270208÷ ž™§K€­5‚(:õ †ÌöÒ’ã¹Ë disk.rbd0.out‚0& disk.nvme0c0n1.out‚100676.29296875$ disk.nvme0n1.out‚100676.29296875& disk.nvme1n1.out‚48126.34521484375$ disk.dm-0.out‚131755.21533203125 disk.dm-1.in‚0! disk.dm-1.out‚15583.509765625( proc.memory.availableMB‚ 482.31640625 network.recv‚ 4.006471e+06 disk.dm-0.in‚ 539.2421875$ memory_percent‚97.05617427825928 cpu‚25.697942021819546 network.sent‚ 2.7281405e+07) disk./.usagePercent‚76.22122442347668" disk.nvme0c0n1.in‚ 309.37109375 disk.md0.in‚0 disk.md0.out‚ 700.29296875" proc.memory.rssMB‚ 3025.7734375) proc.memory.percent‚18.46785545349121% disk./.usageGB‚1170.1863479614258 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚48126.34521484375 disk.nvme1n1.in‚ 229.87109375 proc.cpu.threads‚17ÛÝ?0ž®5‚(:“ †ÌöÒṏÌ gpu.0.gpu‚64.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚275.478' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.82600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘¿¶4¯5/ _wandbruntime‚15420  _runtime‚15420V;‰9'°5N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137650632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2587197076132096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚565174324035584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137650634 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19749034943488 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168221545 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40577558 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168211388 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137650634 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305872179 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305862020 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2606916335743973 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚565174305423360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305872179 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70235777 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚585033868644864 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19859544609280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305862020 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚16999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2622177522284544 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2602428828772352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19859430958592 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2602428828772352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2602428845811712 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19749034943488 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29658219 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19748693512192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚15420.228080161 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19719242572517 _step‚1400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19748693512192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7845222466522176e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168211388 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2587197093171456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2606915978143227 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19748693512192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 _runtime‚15420.228080161 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19719242572517 _step‚1400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19748693512192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7845222466522176e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168211388 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2587197093171456 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2606915978143227 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137650634 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305872179 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305862020 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2606916335743973 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚565174305423360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305872179 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70235777 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚585033868644864 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3ø ê§.4²5/ _wandbruntime‚15420  _runtime‚15420LI¸E³5‹ 0loss/FRt2/assignment_loss‚0.6274131072070404 /loss/FRt2/detection_loss‚1.4571313586365662 1loss/SRqqt1/detection_loss‚0.6404054551902745 %loss/total_loss‚7.152336685754392 _runtime‚15420.228828172 2loss/FBt1/assignment_loss‚0.008390366459521736 3loss/FBt2/assignment_loss‚5.408822081628036e-06 /loss/FRt1/detection_loss‚0.4830524791772187 1loss/SRqqt2/detection_loss‚0.6314524917158413 /loss/FBt2/detection_loss‚0.3791518418117145 trainer/global_step‚16999 % _timestamp‚1.7845222466534774e+09 _step‚1401 0loss/FRt1/assignment_loss‚1.6006019629241195 3loss/SRqqt1/assignment_loss‚0.44292484802670073 3loss/SRqqt2/assignment_loss‚0.42339044435334233 0loss/FBt1/detection_loss‚0.45841692142997037 epoch‚42ýªŽŠ´5 1loss/SRqqt1/detection_loss‚0.6404054551902745 %loss/total_loss‚7.152336685754392 _runtime‚15420.228828172 2loss/FBt1/assignment_loss‚0.008390366459521736 3loss/FBt2/assignment_loss‚5.408822081628036e-06 /loss/FRt1/detection_loss‚0.4830524791772187 1loss/SRqqt2/detection_loss‚0.6314524917158413 /loss/FBt2/detection_loss‚0.3791518418117145 trainer/global_step‚16999 % _timestamp‚1.7845222466534774e+09 _step‚1401 0loss/FRt1/assignment_loss‚1.6006019629241195 3loss/SRqqt1/assignment_loss‚0.44292484802670073 3loss/SRqqt2/assignment_loss‚0.42339044435334233 0loss/FBt1/detection_loss‚0.45841692142997037 epoch‚42 0loss/FRt2/assignment_loss‚0.6274131072070404 /loss/FRt2/detection_loss‚1.4571313586365662ù 8yÉgƒµ5‚(:ø •ÌöÒ’ò­Ë) disk./.usagePercent‚76.22123063532698" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0n1.out‚100678.3564453125& disk.nvme1n1.out‚48127.10791015625$ disk.dm-0.out‚131757.19580078125 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚97.05617427825928% disk./.usageGB‚1170.1864433288574 disk.md0.out‚ 700.6484375( proc.memory.availableMB‚ 482.31640625" proc.memory.rssMB‚ 3025.7734375) proc.memory.percent‚18.46785545349121 cpu‚25.697685467546908 network.sent‚ 2.7333118e+07( disk.nvme0c0n1.out‚100678.3564453125 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚48127.10791015625! disk.dm-1.out‚15583.587890625 network.recv‚ 4.011494e+06ÐÚ Zž¶5‚(:’ •ÌöÒÚê¦Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320W¡dõÿ gpu.0.temp‚56.0 gpu.0.powerWatts‚283.55' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.51666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w‰èÒ‚·5‚(:÷ ¤ÌöÒˆ·ÜË disk.md0.out‚ 701.03125$ disk.dm-0.out‚131757.83251953125 disk.rbd0.out‚0 cpu‚25.697476298064448 network.recv‚ 4.016496e+06 disk.md0.in‚0 proc.cpu.threads‚17 network.sent‚ 2.7387105e+07) disk./.usagePercent‚76.22122318110662' disk.nvme0n1.out‚100679.29638671875" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.05486297607422% proc.memory.availableMB‚ 482.53125" proc.memory.rssMB‚ 3025.7734375 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0# disk.dm-1.out‚15583.71923828125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375) disk.nvme0c0n1.out‚100679.29638671875) proc.memory.percent‚18.46785545349121% disk./.usageGB‚1170.1863288879395 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚48127.75341796875& disk.nvme1n1.out‚48127.75341796875n¦®`“¸5‚(:ˆ ¤ÌöÒê“ Ì gpu.0.gpu‚40.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚284.637' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.879 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—Lѹ5‚(:ö ³ÌöÒü°ÐË disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚48128.5166015625 disk.md0.in‚0) disk.nvme0c0n1.out‚100681.68896484375 disk.nvme0n1.in‚ 309.37109375 disk.dm-0.in‚ 539.2421875) disk./.usagePercent‚76.22123709565129# disk.dm-1.out‚15583.79345703125 disk.rbd0.out‚0$ memory_percent‚97.05638885498047 cpu‚25.697220700268833 network.sent‚ 2.7390027e+07 network.recv‚ 4.019683e+06% disk./.usageGB‚1170.1865425109863" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚48128.5166015625# disk.dm-0.out‚131759.8408203125" proc.memory.rssMB‚ 3025.7734375' disk.nvme0n1.out‚100681.68896484375 disk.md0.out‚ 701.5390625 disk.dm-1.in‚0 disk.rbd0.in‚0% proc.memory.availableMB‚ 482.28125) proc.memory.percent‚18.46785545349121 proc.cpu.threads‚17£p°žº5‚(:“ ³ÌöÒ²µ³Ì gpu.0.gpu‚94.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.446' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.14866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 òÖø4»5/ _wandbruntime‚15465  _runtime‚15465 ÿws˼5Å _runtime‚15465.083027788 _step‚1402 ' lr-AdamW/pg1‚3.3734027837519364e-05 ' lr-AdamW/pg2‚3.3734027837519364e-05 trainer/global_step‚17049 $ _timestamp‚1.784522291506671e+09‹•¥Rн5Ê trainer/global_step‚17049 $ _timestamp‚1.784522291506671e+09 _runtime‚15465.083027788 _step‚1402 ' lr-AdamW/pg1‚3.3734027837519364e-05 ' lr-AdamW/pg2‚3.3734027837519364e-05ú WÃñ¬4¾5/ _wandbruntime‚15465  _runtime‚15465+fÓ (¿5†P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19803152515072 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2629497431892480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29743188 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168696527 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2629497790355968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2614194455674511 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚17049 % _timestamp‚1.7845222915074494e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2609694296416768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 306718873 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 306729018 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2614194813275145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚15465.083099348 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70432440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19803493939200 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168686384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2609694279377408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29743185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40689255%äÚï(À5‹P RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19803152515072 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2629497431892480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29743188 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168696527 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2629497790355968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2614194455674511 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 trainer/global_step‚17049 % _timestamp‚1.7845222915074494e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2609694296416768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 306718873 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 306729018 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2614194813275145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚15465.083099348 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70432440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19803493939200 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168686384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2609694279377408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29743185 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40689255 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0û Q•T94Á5/ _wandbruntime‚15465  _runtime‚15465™p¶["Â5N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138040284 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2609842570918912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2594568609505696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2629646816493568 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138040284 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138040282 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29744922 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29744919 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚586748708200448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚15465.712708179 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 306736322 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138040282 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40691480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19915370660864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚17049 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2629647174964224 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚566833337539584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2609842570918912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168696040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168696040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70436399 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19915267495936 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168706197 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19804262614016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2614343352330981 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚586748586423296 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2609842553879552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚1404 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306746481 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168706197 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70436670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784522292136908e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 306746481Sž%m'Ã5”N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2614343352330981 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚586748586423296 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2609842553879552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚1404 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306746481 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168706197 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70436670 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784522292136908e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 306746481 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138040284 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2609842570918912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2594568609505696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2629646816493568 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138040284 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138040282 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29744922 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29744919 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚586748708200448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚15465.712708179 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 306736322 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138040282 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40691480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19915370660864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 trainer/global_step‚17049 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2629647174964224 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚566833337539584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2609842570918912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168696040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168696040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70436399 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19915267495936 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168706197 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19804262614016 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928ü ¤sm4Ä5/ _wandbruntime‚15465  _runtime‚15465æc ¹Å5ˆ 3loss/FBt2/assignment_loss‚7.012781203923036e-05 /loss/FRt2/detection_loss‚1.4592639277544364 %loss/total_loss‚6.985259059593944 3loss/SRqqt1/assignment_loss‚0.40943870898024814 epoch‚42 % _timestamp‚1.7845222921379995e+09 _step‚1405 0loss/FRt1/assignment_loss‚1.5670696548316971 0loss/FRt2/assignment_loss‚0.5674418562383794 2loss/FBt1/assignment_loss‚0.015471844964064018 1loss/SRqqt1/detection_loss‚0.6413384342634856 0loss/SRqqt2/detection_loss‚0.619295756436849 0loss/FBt2/detection_loss‚0.41217572827617704 /loss/FRt1/detection_loss‚0.4229266128793381 /loss/FBt1/detection_loss‚0.4179632410492139 trainer/global_step‚17049 _runtime‚15465.71326934 2loss/SRqqt2/assignment_loss‚0.4528031661080165ä êÆ5 2loss/SRqqt2/assignment_loss‚0.4528031661080165 3loss/FBt2/assignment_loss‚7.012781203923036e-05 /loss/FRt2/detection_loss‚1.4592639277544364 %loss/total_loss‚6.985259059593944 3loss/SRqqt1/assignment_loss‚0.40943870898024814 epoch‚42 % _timestamp‚1.7845222921379995e+09 _step‚1405 0loss/FRt1/assignment_loss‚1.5670696548316971 0loss/FRt2/assignment_loss‚0.5674418562383794 2loss/FBt1/assignment_loss‚0.015471844964064018 1loss/SRqqt1/detection_loss‚0.6413384342634856 0loss/SRqqt2/detection_loss‚0.619295756436849 0loss/FBt2/detection_loss‚0.41217572827617704 /loss/FRt1/detection_loss‚0.4229266128793381 /loss/FBt1/detection_loss‚0.4179632410492139 trainer/global_step‚17049 _runtime‚15465.71326934ý ˜hÕ…Ç5‚(:ú ÂÌöÒ¼ÍàË" disk.nvme1c1n1.in‚ 229.87109375 network.sent‚ 2.7408958e+07) disk./.usagePercent‚76.22126840337677" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚48129.06591796875) proc.memory.percent‚18.46785545349121 proc.cpu.threads‚17% disk./.usageGB‚1170.1870231628418" disk.dm-0.out‚131760.978515625 disk.dm-1.in‚0) disk.nvme0c0n1.out‚100683.04931640625 disk.nvme1n1.in‚ 229.87109375$ memory_percent‚97.05698490142822( proc.memory.availableMB‚ 482.18359375 network.recv‚ 4.023689e+06 disk.md0.in‚0 disk.rbd0.in‚0' disk.nvme0n1.out‚100683.04931640625& disk.nvme1n1.out‚48129.06591796875" proc.memory.rssMB‚ 3025.7734375 cpu‚25.697013213081505 disk.md0.out‚ 701.82421875 disk.dm-0.in‚ 539.2421875" disk.dm-1.out‚15583.9404296875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375è%w“È5‚(:ˆ ÂÌöÒ°Î¥Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.209' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚95.403 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ %„É5‚(:ù ÑÌöÒعÔË) disk./.usagePercent‚76.22126567016264 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 network.sent‚ 2.7459191e+07 network.recv‚ 4.02798e+06 disk.dm-0.in‚ 539.2421875# disk.dm-0.out‚131765.7353515625" disk.dm-1.out‚15584.0068359375) disk.nvme0c0n1.out‚100687.42041015625 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚48130.06689453125% disk./.usageGB‚1170.1869812011719 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375" proc.memory.rssMB‚ 3025.7734375) proc.memory.percent‚18.46785545349121 proc.cpu.threads‚17 disk.md0.out‚ 702.06640625' disk.nvme0n1.out‚100687.42041015625( disk.nvme1c1n1.out‚48130.06689453125$ memory_percent‚97.05555438995361( proc.memory.availableMB‚ 482.41796875 cpu‚25.69714537905115ËàžÊ5‚(:“ ÑÌöÒ–í˜Ì gpu.0.gpu‚91.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.108' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.36933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™˜™€Ë5‚(:õ àÌöÒ ÝÌË' disk.nvme0n1.out‚100689.35595703125 network.sent‚ 2.7461565e+07 disk.rbd0.in‚0) disk.nvme0c0n1.out‚100689.35595703125 disk.md0.out‚ 702.40625 disk.dm-0.in‚ 539.2421875" proc.memory.rssMB‚ 3025.7734375) proc.memory.percent‚18.46785545349121 network.recv‚ 4.031329e+06" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0( proc.memory.availableMB‚ 482.89453125 cpu‚25.69745427357303) disk./.usagePercent‚76.22130542600452% disk./.usageGB‚1170.1875915527344 disk.dm-1.in‚0" disk.dm-1.out‚15584.1533203125 disk.rbd0.out‚0& disk.nvme1n1.out‚48130.82568359375! disk.dm-0.out‚131767.52734375( disk.nvme1c1n1.out‚48130.82568359375$ memory_percent‚97.05264568328857 proc.cpu.threads‚17ˆç žÌ5‚(:“ àÌöÒ®Û¢Ì gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚268.036' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.34533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„<²å4Í5/ _wandbruntime‚15511  _runtime‚15511çÝòÊÎ5Ä % _timestamp‚1.7845223377561543e+09 _runtime‚15511.331820814 _step‚1406 & lr-AdamW/pg1‚3.262107738114959e-05 & lr-AdamW/pg2‚3.262107738114959e-05 trainer/global_step‚17099ª=pzÏÏ5É _step‚1406 & lr-AdamW/pg1‚3.262107738114959e-05 & lr-AdamW/pg2‚3.262107738114959e-05 trainer/global_step‚17099 % _timestamp‚1.7845223377561543e+09 _runtime‚15511.331820814þ ÿ#C4Ð5/ _wandbruntime‚15511  _runtime‚15511°ÿ6 (Ñ5…P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138422141 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2617108004484608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _runtime‚15511.334283268 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19858721616896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169171036 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2621621829862153 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚588429093884416 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2601792729251184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40803377 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 $ _timestamp‚1.784522337758572e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70633556 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29829885 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19969941723136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169181179 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2617108021523968 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2636966726101504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588429224057344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19828760049695 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚568459170773504 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2636967084564992 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚17099 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2621621472261519 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2636967084564992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169171036 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138422139 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40803668 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2601792712211824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2617108021523968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2617108004484608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚568459152161280 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19829100610969 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19859063041024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19858721616896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307593175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2636966726101504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138422141Ûzï ¿Ò5ŠP N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138422139 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19970053283840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚1407 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19858721616896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307593175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2636966726101504 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138422141 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138422141 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2617108004484608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _runtime‚15511.334283268 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19858721616896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169171036 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2621621829862153 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚588429093884416 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2601792729251184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40803377 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 $ _timestamp‚1.784522337758572e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70633556 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29829885 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19969941723136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169181179 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2617108021523968 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2636966726101504 ]ODeviceStatsMonitor.on_train_batch_starËÎèÐQt/inactive_split_bytes.small_pool.current‚ 111560704 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588429224057344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19828760049695 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚568459170773504 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2636967084564992 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 trainer/global_step‚17099 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2621621472261519 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2636967084564992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169171036 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138422139 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40803668 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2601792712211824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2617108021523968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2617108004484608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚568459152161280 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19829100610969 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19859063041024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2617256278986752 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2601940142879296 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169190849 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚588463394680320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2621770011317243 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2637116469173248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2621770368917989 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 307610624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70637262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29831622 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40805923 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169180692 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138429934 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚588463520651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307610624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138429932 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19859831715840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138429934 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2637116110702592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70637545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19829868437947 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚17099 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29831619 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307620783 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2617256296026112 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169190849 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307620783 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19971169608192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2601940159918656 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚15511.965088041 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚568492351043584 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19859831715840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19971062248960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚07) 'Õ5•N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169190849 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚588463394680320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2621770011317243 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2637116469173248 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2621770368917989 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 307610624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70637262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29831622 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40805923 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169180692 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138429934 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚588463520651776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307610624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138429932 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19859831715840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138429934 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2637116110702592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70637545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19829868437947 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 trainer/global_step‚17099 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29831619 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307620783 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2617256296026112 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169190849 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307620783 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19971169608192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2601940159918656 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 _runtime‚15511.965088041 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚568492351043584 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19859831715840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19971062248960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2617256296026112 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚19830208999333 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2637116469173248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7845223383892214e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2617256278986752 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2601940142879296 EÎá5È & lr-AdamW/pg2‚3.152576548758699e-05 trainer/global_step‚17149 $ _timestamp‚1.784522383387691e+09 _runtime‚15556.963294501 _step‚1410 & lr-AdamW/pg1‚3.152576548758699e-05‚ j)}ý4â5/ _wandbruntime‚15556  _runtime‚15556Rì”q6ã5„P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2624521746631168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19914632142848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20025866636288 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2629048488848527 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2609164262624784 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2609164245585424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19884243263103 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138811789 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.’/àgÔ"large_pool.freed‚2624521729591808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2644436020310528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169665831 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚17149 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2624521729591808 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19914290718720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70834322 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19884583824377 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2644436378774016 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169655688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚590144050913792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚570118165665280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138811791 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 $ _timestamp‚1.784522383388679e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308477622 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _runtime‚15556.964196202 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19914290718720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚1411 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2629048846449161 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29916588 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138811791 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2644436020310528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169665831 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70834038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚590143933323776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138811789 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308467477 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29916585 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚570118184277504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20025767658496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169655688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40917734 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2644436378774016[«'(ä5‰P M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70834038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚590143933323776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138811789 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308467477 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29916585 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚570118184277504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20025767658496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169655688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40917734 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2644436378774016 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2624521746631168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19914632142848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20025866636288 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2629048488848527 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2609164262624784 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2609164245585424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚19884243263103 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138811789 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2624521729591808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2644436020310528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 169665831 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚17149 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2624521729591808 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚19914290718720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70834322 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19884583824377 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2644436378774016 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169655688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚590144050913792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚570118165665280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138811791 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 $ _timestamp‚1.784522383388679e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308477622 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 _runtime‚15556.964196202 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚19914290718720 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚1411 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2629048846449161 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29916588 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138811791 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2644436020310528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169665831 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ƒ XJ‹4å5/ _wandbruntime‚15557  _runtime‚15557ž¶I'æ5N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138819584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169665344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19915742248960 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚19915400817664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚570151364547584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138819584 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169665344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚590178350193664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 308484926 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19915742248960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19915400817664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2629197027904251 trainer/global_step‚17149 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 308484926 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169675501 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40919992 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 % _timestamp‚1.7845223843358765e+09 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2609311693292256 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2609311676252896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2624670004093952 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2644585763382272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70838314 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169675501 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20026880384000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19885351651355 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2624670021133312 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2624670004093952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308495085 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚570151345935360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70838021 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308495085 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2644585763382272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20026985646080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2624670021133312 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚19915742248960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚19915400817664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2629197027904251 trainer/global_step‚17149 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 308484926 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169675501 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚290 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40919992 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 % _timestamp‚1.7845223843358765e+09 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2609311693292256 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceߘûkStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2609311676252896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2624670004093952 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2644585763382272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70838314 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169675501 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20026880384000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚19885351651355 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2624670021133312 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2624670004093952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308495085 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚570151345935360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70838021 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308495085 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2644585763382272 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20026985646080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2624670021133312 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2651905672983040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19970201244672 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1415 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚571777179169280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2631935454699008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚15601.548762722 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19970201244672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20081588894208 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19940067037785 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41031925 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2651905314519552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20081698357760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 % _timestamp‚1.7845224279731286e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2651905672983040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚591858896139264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30003288 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2631935471738368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚17199 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚591858768063488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30003285 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139201439 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2631935471738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309351924 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139201441 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2651905314519552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71035213 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚15601.548762722 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚19970201244672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20081588894208 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚19940067037785 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41031925 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2651905314519552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20081698357760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 % _timestamp‚1.7845224279731286e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2651905672983040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚591858896139264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30003288 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2631935471738368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 trainer/global_step‚17199 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚591858768063488 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚301 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30003285 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139201439 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2631935471738368 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 309351924 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139201441 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2651905314519552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71035213 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_startQÖCÁ¡/active_bytes.all.allocated‚2651905672983040 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚19970201244672 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚1415‡ ]gë4÷5/ _wandbruntime‚15602  _runtime‚156022äMz'ø5N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚571810359439360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139209232 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2632083746240512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71039263 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2636624402092005 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20082816861696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20082703211008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309359228 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2632083746240512 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170160153 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2616683226665856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2632083729201152 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2632083729201152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41034241 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71038980 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571810378051584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309369387 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30005019 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚591893062650368 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170149996 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2652055057591296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170160153 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19971311350784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309369387 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _runtime‚15602.171536719 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139209234 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309359228 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170149996 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139209234e²æò'ù5•N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139209234 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309359228 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170149996 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139209234 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚571810359439360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139209232 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2632083746240512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71039263 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2636624402092005 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20082816861696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20082703211008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309359228 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2632083746240512 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170160153 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2616683226665856 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2632083729201152 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2632083729201152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41034241 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71038980 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571810378051584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309369387 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30005019 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚591893062650368 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170149996 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2652055057591296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170160153 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚19971311350784 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309369387 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _runtime‚15602.171536719 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ˆ ïX4ú5/ _wandbruntime‚15602  _runtime‚15602G¥Ä û5‡ 2loss/SRqqt1/assignment_loss‚0.4448482660922478 .loss/FRt1/detection_loss‚0.445552953176201 % _timestamp‚1.7845224285967672e+09 _runtime‚15602.172028349 /loss/FRt2/assignment_loss‚0.662706185328333 3loss/FBt1/assignment_loss‚0.0006463981249468919 /loss/FBt2/detection_loss‚0.3857818818997909 epoch‚42 3loss/FBt2/assignment_loss‚0.0004791453527429483 /loss/FRt2/detection_loss‚1.5223640897589776 1loss/SRqqt2/detection_loss‚0.6327736314964544 0loss/FBt1/detection_loss‚0.25973957638834977 %loss/total_loss‚6.937613612886958 0loss/FRt1/assignment_loss‚1.5052503661992525 2loss/SRqqt2/assignment_loss‚0.4864190632034198 0loss/SRqqt1/detection_loss‚0.591052055866241 trainer/global_step‚17199 _step‚1417uQYsü5Œ 0loss/SRqqt1/detection_loss‚0.591052055866241 trainer/global_step‚17199 _step‚1417 2loss/SRqqt1/assignment_loss‚0.4448482660922478 .loss/FRt1/detection_loss‚0.445552953176201 % _timestamp‚1.7845224285967672e+09 _runtime‚15602.172028349 /loss/FRt2/assignment_loss‚0.662706185328333 3loss/FBt1/assignment_loss‚0.0006463981249468919 /loss/FBt2/detection_loss‚0.3857818818997909 epoch‚42 3loss/FBt2/assignment_loss‚0.0004791453527429483 /loss/FRt2/detection_loss‚1.5223640897589776 1loss/SRqqt2/detection_loss‚0.6327736314964544 0loss/FBt1/detection_loss‚0.25973957638834977 %loss/total_loss‚6.937613612886958 0loss/FRt1/assignment_loss‚1.5052503661992525 2loss/SRqqt2/assignment_loss‚0.4864190632034198‰ \•8ä~ý5‚(:ó ÉÍöÒË—òË$ disk.nvme1n1.out‚48138.419921875 disk.md0.in‚0 disk.md0.out‚ 704.78125$ disk.dm-0.out‚131785.54833984375" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0c0n1.out‚100707.447265625( proc.memory.availableMB‚ 482.49609375 network.recv‚ 4.058441e+06) disk./.usagePercent‚76.04421750087144$ disk./.usageGB‚1167.468849182129 disk.rbd0.out‚0# proc.memory.rssMB‚ 3025.80078125 network.sent‚ 2.7623807e+07 disk.nvme0n1.in‚ 309.37109375 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15586.63818359375 disk.rbd0.in‚0$ memory_percent‚97.05507755279541* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17% disk.nvme0n1.out‚100707.447265625" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1c1n1.out‚48138.419921875 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0 cpu‚25.700077320670665ã\„‘þ5‚(:† ÉÍöÒƒ­ºÌ gpu.0.gpu‚77.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.16' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.72 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎÙ(Ë4ÿ5/ _wandbruntime‚15619  _runtime‚15619ÀÜ”/$€6¸z 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992322325706482 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29813477396965027 /particle/accuracy_4_of_4‚0.9462108612060547 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8858659863471985 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4592320919036865 +jet/accuracy_4_of_4‚0.01670061983168125 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6036827564239502 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6263099908828735 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34558990597724915 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7519387602806091 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6957562565803528 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.227886363863945 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3352542817592621 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5522961616516113 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5627285838127136 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 particle/accuracy_1_of_2‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6752349138259888 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06300872564315796 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1226620003581047 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.340911328792572 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45911163091659546 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6732192039489746 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.050164029002189636 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2753429710865021 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 6validation_average_jet_accuracy‚0.3670659065246582 particle/accuracy_3_of_5‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7175061106681824 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5836741328239441 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.577263593673706 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6386914849281311 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30924686789512634 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7227694392204285 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08660486340522766 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6354916095733643 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2833806872367859 +particle/sensitivity‚0.6406649351119995 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5364583134651184 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9481417536735535 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4845624566078186 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07990758866071701 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28987276554107666 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5128299593925476 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6997857689857483 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.038548968732357025 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2987890839576721 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23242704570293427 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.280870646238327 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22884757816791534 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5818579196929932 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5836643576622009 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚1418 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.630845308303833 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8573317527770996 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5841352939605713 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 (jet/accuracy_1_of_2‚0.76038658618927 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_3‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9109980463981628 particle/accuracy_2_of_5‚1 epoch‚42 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6084664463996887 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26476770639419556 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_1_of_4‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6686263084411621 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28902897238731384 particle/accuracy_1_of_1‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2178487479686737 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 'particle/f_score‚0.6516990661621094 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988906383514404 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6020833253860474 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33561092615127563 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7302812337875366 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9903878569602966 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.584753155708313 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.22666668891906738 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 +jet/accuracy_3_of_4‚0.09184560924768448 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8532690405845642 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08593308180570602 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29993700981140137 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30009031295776367 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Pur°ŠB'ity/2FRt*SRqqt0FBt/FRt_purity‚0.2571898102760315 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 jet/accuracy_2_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3199833035469055 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7519387602806091 _runtime‚15619.662190942 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6117640733718872 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0857299193739891 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2936900854110718 *jet/accuracy_1_of_3‚0.9134015440940857 jet/accuracy_3_of_5‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07990758866071701 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30182820558547974 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4458905756473541 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9971833229064941 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2639480233192444 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48449787497520447 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9107228517532349 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2411135882139206 % _timestamp‚1.7845224460860415e+09 )jet/accuracy_2_of_2‚0.158140167593956 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.700772762298584 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9967963695526123 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.051702845841646194 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6723023653030396 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5635538697242737 jet/accuracy_5_of_5‚0 trainer/global_step‚17199 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2202112078666687 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8720710277557373 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19947364926338196 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8693861365318298 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6188721656799316 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6324947476387024 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6093202233314514 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5661154985427856 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 +jet/accuracy_1_of_1‚0.32663723826408386 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7574769258499146 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6503392457962036 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6909865140914917 )jet/accuracy_2_of_3‚0.551274299621582 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5852392315864563 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2032419890165329 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8633031249046326 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6079872846603394 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6881189942359924 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7526041865348816 *jet/accuracy_3_of_3‚0.0761389210820198 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6641937494277954 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_4‚0.9613099694252014 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8529419898986816 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9907387495040894 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6578851938247681 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7131251096725464 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )loss/val_total_loss‚7.437181426697154 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2755625545978546 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1006896123290062 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5635538697242737 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 /particle/accuracy_3_of_3‚0.9994120597839355 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08533727377653122 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28902897238731384 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2735154926776886u‹C=6½z 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6686263084411621 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28902897238731384 particle/accuracy_1_of_1‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2178487479686737 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 'particle/f_score‚0.6516990661621094 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988906383514404 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.6020833253860474 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33561092615127563 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7302812337875366 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9903878569602966 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.584753155708313 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.22666668891906738 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 +jet/accuracy_3_of_4‚0.09184560924768448 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8532690405845642 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08593308180570602 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29993700981140137 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30009031295776367 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2571898102760315 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 jet/accuracy_2_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3199833035469055 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7519387602806091 _runtime‚15619.662190942 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6117640733718872 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0857299193739891 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2936900854110718 *jet/accuracy_1_of_3‚0.9134015440940857 jet/accuracy_3_of_5‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07990758866071701 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30182820558547974 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4458905756473541 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9971833229064941 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2639480233192444 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48449787497520447 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9107228517532349 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2411135882139206 % _timestamp‚1.7845224460860415e+09 )jet/accuracy_2_of_2‚0.158140167593956 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.700772762298584 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9967963695526123 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.051702845841646194 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6723023653030396 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5635538697242737 jet/accuracy_5_of_5‚0 trainer/global_step‚17199 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2202112078666687 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_2‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8720710277557373 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19947364926338196 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8693861365318298 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6188721656799316 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6324947476387024 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6093202233314514 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5661154985427856 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 +jet/accuracy_1_of_1‚0.32663723826408386 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7574769258499146 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6503392457962036 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6909865140914917 )jet/accuracy_2_of_3‚0.551274299621582 jet/accuracy_4_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5852392315864563 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2032419890165329 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8633031249046326 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6079872846603394 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6881189942359924 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7526041865348816 *jet/accuracy_3_of_3‚0.0761389210820198 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6641937494277954 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_4‚0.9613099694252014 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8529419898986816 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9907387495040894 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6578851938247681 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7131251096725464 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 )loss/val_total_loss‚7.437181426697154 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2755625545978546 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.1006896123290062 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_5‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5635538697242737 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 /particle/accuracy_3_of_3‚0.9994120597839355 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08533727377653122 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28902897238731384 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2735154926776886 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992322325706482 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29813477396965027 /particle/accuracy_4_of_4‚0.9462108612060547 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8858659863471985 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4592320919036865 +jet/accuracy_4_of_4‚0.01670061983168125 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6036827564239502 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6263099908828735 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34558990597724915 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7519387602806091 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6957562565803528 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.227886363863945 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3352542817592621 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5522961616516113 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5627285838127136 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 particle/accuracy_1_of_2‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6752349138259888 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06300872564315796 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1226620003581047 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.340911328792572 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45911163091659546 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6732192039489746 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.050164029002189636 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2753429710865021 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_4‚1 6validation_average_jet_accuracy‚0.3670659065246582 particle/accuracy_3_of_5‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7175061106681824 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5836741328239441 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.577263593673706 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6386914849281311 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30924686789512634 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7227694392204285 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08660486340522766 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6354916095733643 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2833806872367859 +particle/sensitivity‚0.6406649351119995 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.5364583134651184 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9481417536735535 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4845624566078186 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07990758866071701 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28987276554107666 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5128299593925476 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6997857689857483 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.038548968732357025 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2987890839576721 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23242704570293427 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.280870646238327 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9952875971794128 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22884757816791534 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5818579196929932 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5836643576622009 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚1418 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.630845308303833 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8573317527770996 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5841352939605713 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 (jet/accuracy_1_of_2‚0.76038658618927 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_3‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9109980463981628 particle/accuracy_2_of_5‚1 epoch‚42 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6084664463996887 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26476770639419556 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_1_of_4‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127Š @öSĆ‚6‚(:û ØÍöÒ—çãË network.recv‚ 4.063582e+06% disk./.usageGB‚1170.2305603027344 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375) disk.nvme0c0n1.out‚102621.85498046875( proc.memory.availableMB‚ 482.14453125# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375 proc.cpu.threads‚17) disk./.usagePercent‚76.22410423727304 disk.dm-1.in‚0' disk.nvme0n1.out‚102621.85498046875" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚48138.9619140625 disk.md0.out‚ 704.89453125 cpu‚25.701180144060583 network.sent‚ 2.768624e+07# disk.dm-1.out‚15586.76416015625 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚48138.9619140625 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875# disk.dm-0.out‚133700.1025390625$ memory_percent‚97.05722332000732¯¸>垃6‚(:“ ØÍöÒ‹«Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚121.345' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.44833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ýÂÏ„6‚(:ô çÍöÒر˜Ë network.sent‚ 2.7730261e+07) disk./.usagePercent‚76.22412113350585% disk./.usageGB‚1170.2308197021484 disk.nvme1n1.in‚ 229.87109375* proc.memory.percent‚18.468022346496582 cpu‚25.701500305771336 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚97.05712795257568# proc.memory.rssMB‚ 3025.80078125" disk.nvme1n1.out‚ 48143.0703125 disk.md0.out‚ 705.3359375 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15590.00634765625 disk.rbd0.in‚0) disk.nvme0c0n1.out‚102625.64013671875 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 network.recv‚ 4.068196e+06" disk.dm-0.out‚133703.794921875" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0n1.out‚102625.64013671875$ disk.nvme1c1n1.out‚ 48143.0703125( proc.memory.availableMB‚ 482.16015625 proc.cpu.threads‚17ß"ª’…6‚(:‡ çÍöÒá…ñË gpu.0.gpu‚22.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚53.0 gpu.0.powerWatts‚119.421' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.807 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä2Ÿs†6‚(:è öÍöÒóÛÑË' proc.memory.availableMB‚ 482.1640625# proc.memory.rssMB‚ 3025.80078125" disk.nvme0c0n1.in‚ 309.37109375% disk.nvme0c0n1.out‚103349.0078125# disk.nvme1c1n1.out‚ 48266.265625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0% disk./.usageGB‚1170.1894493103027 disk.nvme0n1.in‚ 309.37109375# disk.nvme0n1.out‚103349.0078125" disk.nvme1c1n1.in‚ 229.87109375! disk.nvme1n1.out‚ 48266.265625 disk.dm-0.in‚ 539.2421875 proc.cpu.threads‚17 network.recv‚ 4.071722e+06) disk./.usagePercent‚76.22142643284825" disk.dm-0.out‚134549.755859375! disk.dm-1.out‚15590.138671875 disk.rbd0.out‚0* proc.memory.percent‚18.468022346496582 cpu‚25.701848180164372 network.sent‚ 2.7737017e+07 disk.md0.out‚ 705.55078125$ memory_percent‚97.05710411071777‡˜œ‡6‚(:‘ öÍöÒ®ÁœÌ gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚286.595' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.53166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ÉÈú4ˆ6/ _wandbruntime‚15665  _runtime‚15665j4Ó̉6Æ % _timestamp‚1.7845224914599946e+09 _runtime‚15665.035601938 _step‚1419 ' lr-AdamW/pg1‚2.9388337650818066e-05 ' lr-AdamW/pg2‚2.9388337650818066e-05 trainer/global_step‚17249ú™pnÑŠ6Ë _step‚1419 ' lr-AdamW/pg1‚2.9388337650818066e-05 ' lr-AdamW/pg2‚2.9388337650818066e-05 trainer/global_step‚17249 % _timestamp‚1.7845224914599946e+09 _runtime‚15665.035601938‹ ÒÛú4‹6/ _wandbruntime‚15665  _runtime‚15665±ÊÓ¹òŒ6†P QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170706486 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170716629 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚573458789947904 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20046088182272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139713899 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139713899 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2641291540721152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170716629 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2641291540721152 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚17249 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310420383 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30094104 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2661337270439936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 310430528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170706486 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2661337270439936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2625828568340848 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2661337628903424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚593616911437824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/rÌ´Ëeserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2661337628903424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20045746758144 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41180962 _step‚1420 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139713897 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚573458771335680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7845224914610896e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71275066 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30094101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41180672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2641291523681792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20046088182272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20158140102144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170716629 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚573458789947904 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20046088182272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139713899 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139713899 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2641291540721152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170716629 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2641291540721152 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚17249 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310420383 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30094104 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2661337270439936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 310430528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 170706486 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2661337270439936 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2625828568340848 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2661337628903424 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚593616911437824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2661337628903424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20045746758144 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41180962 _step‚1420 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139713897 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚573458771335680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7845224914610896e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71275066 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30094101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41180672 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2641291523681792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20046088182272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20158140102144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71278832 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20047198288384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41182997 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20046856857088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30095838 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310447991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170716142 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2641439798183936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30095835 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845224920921292e+09 _step‚1421 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20047198288384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139721692 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170726299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2641439815223296 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310437832 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310447991 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20016962488454 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚593651330759168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚17249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚15665.668148402 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139721692 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚573491951605760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41183280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170726299 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528«=óÿh 6•N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30095835 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845224920921292e+09 _step‚1421 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20047198288384 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139721692 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170726299 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2641439815223296 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310437832 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310447991 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20016962488454 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚593651330759168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚17249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚15665.668148402 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139721692 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚573491951605760 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41183280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170726299 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2641439798183936 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71278832 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20047198288384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_siz+ëDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41182997 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20046856857088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30095838 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310447991 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170716142 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2641439798183936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 “¸@Â4‘6/ _wandbruntime‚15665  _runtime‚15665 ¥æ|’6‰ .loss/FRt2/detection_loss‚1.453618952998012 /loss/SRqqt1/detection_loss‚0.65503529102517 0loss/FBt2/detection_loss‚0.37633645654152914 _step‚1422 0loss/FRt2/assignment_loss‚0.7087509298214206 epoch‚43 % _timestamp‚1.7845224920943048e+09 3loss/FBt2/assignment_loss‚4.972529840547766e-06 0loss/FRt1/detection_loss‚0.46543438372003104 1loss/SRqqt2/detection_loss‚0.6038750529324419 trainer/global_step‚17249 _runtime‚15665.669903834 3loss/SRqqt1/assignment_loss‚0.39663374858846717 3loss/SRqqt2/assignment_loss‚0.45072754709100477 0loss/FBt1/detection_loss‚0.36707111291538286 %loss/total_loss‚7.004020670978451 0loss/FRt1/assignment_loss‚1.5190907214909064 1loss/FBt1/assignment_loss‚0.00744150132424443Áë^“6Ž 3loss/FBt2/assignment_loss‚4.972529840547766e-06 0loss/FRt1/detection_loss‚0.46543438372003104 1loss/SRqqt2/detection_loss‚0.6038750529324419 trainer/global_step‚17249 _runtime‚15665.669903834 3loss/SRqqt1/assignment_loss‚0.39663374858846717 3loss/SRqqt2/assignment_loss‚0.45072754709100477 0loss/FBt1/detection_loss‚0.36707111291538286 %loss/total_loss‚7.004020670978451 0loss/FRt1/assignment_loss‚1.5190907214909064 1loss/FBt1/assignment_loss‚0.00744150132424443 .loss/FRt2/detection_loss‚1.453618952998012 /loss/SRqqt1/detection_loss‚0.65503529102517 0loss/FBt2/detection_loss‚0.37633645654152914 _step‚1422 0loss/FRt2/assignment_loss‚0.7087509298214206 epoch‚43 % _timestamp‚1.7845224920943048e+09Ž Ü€”6‚(:õ …ÎöÒ㼃Ì* proc.memory.percent‚18.468022346496582 disk.dm-1.in‚0! disk.dm-1.out‚15590.248046875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375 disk.md0.in‚0( proc.memory.availableMB‚ 482.07421875# proc.memory.rssMB‚ 3025.80078125 network.recv‚ 4.074733e+06) disk./.usagePercent‚76.22110888306122( disk.nvme1c1n1.out‚48269.71533203125 cpu‚25.702163723859215 network.sent‚ 2.7752824e+07% disk./.usageGB‚1170.1845741271973& disk.nvme0c0n1.out‚103378.74609375$ disk.nvme0n1.out‚103378.74609375# memory_percent‚97.0576524734497 proc.cpu.threads‚17 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚48269.71533203125 disk.md0.out‚ 705.9296875 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚134581.99755859375”>F“•6‚(:ˆ …ÎöÒÉ‚ÇÌ gpu.0.gpu‚68.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚278.382' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.794 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«Gun{–6‚(:ð ”ÎöÒýóÇË disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚103379.7158203125" disk.nvme1c1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3025.80078125 disk.nvme0n1.in‚ 309.37109375$ disk.nvme1c1n1.out‚ 48270.4453125* proc.memory.percent‚18.468022346496582$ disk.dm-0.out‚134582.79833984375# disk.dm-1.out‚15590.47314453125& disk.nvme0n1.out‚103379.7158203125 disk.nvme1n1.in‚ 229.87109375# memory_percent‚97.0576286315918 cpu‚25.70244976435984 proc.cpu.threads‚17 network.sent‚ 2.7802449e+07 network.recv‚ 4.079911e+06) disk./.usagePercent‚76.22113298504036$ disk./.usageGB‚1170.184944152832 disk.dm-1.in‚0 disk.rbd0.out‚0" disk.nvme1n1.out‚ 48270.4453125& proc.memory.availableMB‚ 482.078125 disk.md0.in‚0 disk.md0.out‚ 706.24609375 disk.dm-0.in‚ 539.2421875†OÛœ—6‚(:‘ ”ÎöÒ¾×¡Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.485' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.82833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Íõ ˜6‚(:ö £ÎöÒãÍ®Ë disk.md0.out‚ 706.765625 disk.dm-0.in‚ 539.2421875# disk.dm-0.out‚134584.4814453125 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚48271.1943359375 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚48271.1943359375) disk./.usagePercent‚76.22112031286575% disk./.usageGB‚1170.1847496032715 disk.md0.in‚0# memory_percent‚97.0576286315918# proc.memory.rssMB‚ 3025.80078125 cpu‚25.70278205464137" disk.nvme0c0n1.in‚ 309.37109375 network.sent‚ 2.7809337e+07 network.recv‚ 4.083219e+06 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 482.078125 proc.cpu.threads‚17# disk.dm-1.out‚15590.54345703125) disk.nvme0c0n1.out‚103381.81982421875' disk.nvme0n1.out‚103381.81982421875* proc.memory.percent‚18.468022346496582×GžÙ™6‚(:’ £ÎöÒ£ÅþË gpu.0.gpu‚21.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚289.303' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.43433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|Á4š6/ _wandbruntime‚15711  _runtime‚15711i¾F§Ì›6Æ ' lr-AdamW/pg1‚2.8346358990607453e-05 ' lr-AdamW/pg2‚2.8346358990607453e-05 trainer/global_step‚17299 % _timestamp‚1.7845225375863442e+09 _runtime‚15711.161938021 _step‚14236sgÑœ6Ë ' lr-AdamW/pg1‚2.8346358990607453e-05 ' lr-AdamW/pg2‚2.8346358990607453e-05 trainer/global_step‚17299 % _timestamp‚1.7845225375863442e+09 _runtime‚15711.161938021 _step‚1423 F¦46/ _wandbruntime‚15711  _runtime‚15711™ ‹ (ž6†P \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2653271098466672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140103549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140103547 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171201281 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20101315859968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20101315859968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30180804 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30180801 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚575117803451904 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171191138 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2648705265828352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41296556 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚17299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2668806923112448 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2648705248788992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595331785048576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595331913124352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845225375874515e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171201281 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 311304830 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2668806564648960 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _step‚1424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140103547 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20101657284096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140103549 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚575117784839680 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2653271456067306 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2633200101714448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20071337313498 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _runtime‚15711.163052213 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2668806564648960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20214000208896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311294685 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2668806923112448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71477640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20214109672448 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2648705265828352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71477357 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41296836 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171191138 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20070996752224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2648705248788992´C2t(Ÿ6‹P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140103549 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚575117784839680 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2653271456067306 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2633200101714448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20071337313498 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _runtime‚15711.163052213 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2668806564648960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20214000208896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 311294685 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2668806923112448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71477640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20214109672448 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2648705265828352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71477357 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41296836 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171191138 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20070996752224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2648705248788992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2653271098466672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140103549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140103547 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171201281 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20101315859968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20101315859968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30180804 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30180801 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚575117803451904 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171191138 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2648705265828352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41296556 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚17299 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2668806923112448 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2648705248788992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595331785048576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595331913124352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845225375874515e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171201281 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 311304830 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2668806564648960 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _step‚1424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140103547 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20101657284096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 Ë¡ß“4 6/ _wandbruntime‚15711  _runtime‚157110;O>Ú¡6N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20215119368704 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2633347549421280 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140111340 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2653419995123142 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚15711.796164844 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140111342 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20102425958912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 311312134 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311322293 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41298802 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171210951 LDN h!;DeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚1425 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171200794 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171210951 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20102425958912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20102767390208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41299080 trainer/global_step‚17299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71481337 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚575150965109760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 311322293 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311312134 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171200794 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2668956307720704 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2648853523291136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30182535 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20215224630784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595366208352768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575150983721984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20072105140476 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 $ _timestamp‚1.784522538220343e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140111340 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2633347532381920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20072445701862 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚11800Ÿr¢'¢6”N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171210951 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚1425 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171200794 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171210951 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20102425958912 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20102767390208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41299080 trainer/global_step‚17299 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71481337 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚575150965109760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 311322293 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 311312134 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171200794 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2668956307720704 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2648853523291136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30182535 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20215224630784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595366208352768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575150983721984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20072105140476 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 $ _timestamp‚1.784522538220343e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140111340 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2633347532381920 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20072445701862 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20215119368704 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2633347549421280 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140111340 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2653419995123142 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚15711.796164844 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140111342 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20102425958912 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 311312134 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 311322293 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41298802 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0‘ ?³í4£6/ _wandbruntime‚15711  _runtime‚15711¹ÙM¤6‰ 0loss/FRt1/assignment_loss‚1.4292349115268554 2loss/SRqqt1/assignment_loss‚0.4065042712021341 2loss/FBt1/assignment_loss‚0.023386033619770367 1loss/SRqqt1/detection_loss‚0.5683951920516153 trainer/global_step‚17299 _runtime‚15711.797052195 0loss/FRt2/assignment_loss‚0.7660886790071215 2loss/SRqqt2/assignment_loss‚0.4544592271994644 /loss/FRt1/detection_loss‚0.4718931365352556 1loss/SRqqt2/detection_loss‚0.6732210078271385 0loss/FBt2/detection_loss‚0.44139993111800646 %loss/total_loss‚7.136809849388888 2loss/FBt2/assignment_loss‚0.004544241753369946 /loss/FRt2/detection_loss‚1.4606916794531224 0loss/FBt1/detection_loss‚0.43699153809503427 epoch‚43 % _timestamp‚1.7845225382215865e+09 _step‚14263ÊÚ ¥6Ž 2loss/FBt1/assignment_loss‚0.023386033619770367 1loss/SRqqt1/detection_loss‚0.5683951920516153 trainer/global_step‚17299 _runtime‚15711.797052195 0loss/FRt2/assignment_loss‚0.7660886790071215 2loss/SRqqt2/assignment_loss‚0.4544592271994644 /loss/FRt1/detection_loss‚0.4718931365352556 1loss/SRqqt2/detection_loss‚0.6732210078271385 0loss/FBt2/detection_loss‚0.44139993111800646 %loss/total_loss‚7.136809849388888 2loss/FBt2/assignment_loss‚0.004544241753369946 /loss/FRt2/detection_loss‚1.4606916794531224 0loss/FBt1/detection_loss‚0.43699153809503427 epoch‚43 % _timestamp‚1.7845225382215865e+09 _step‚1426 0loss/FRt1/assignment_loss‚1.4292349115268554 2loss/SRqqt1/assignment_loss‚0.4065042712021341’ khŒƒ¦6‚(:ø ²ÎöÒÍßÍ* proc.memory.percent‚18.468022346496582 disk.rbd0.out‚0$ memory_percent‚97.05784320831299 cpu‚25.703078645499833 network.sent‚ 2.7812259e+07% disk./.usageGB‚1170.1856269836426) disk.nvme0c0n1.out‚103383.18701171875& disk.nvme1c1n1.out‚48271.662109375 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚48271.662109375# disk.dm-0.out‚134585.6962890625 disk.dm-1.in‚0 network.recv‚ 4.086931e+06) disk./.usagePercent‚76.22117746188846 disk.md0.in‚0 disk.md0.out‚ 706.984375( proc.memory.availableMB‚ 482.04296875# proc.memory.rssMB‚ 3025.80078125 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚103383.18701171875" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15590.67138671875 disk.rbd0.in‚0§VDž§6‚(:“ ²ÎöÒ›ãèÍ gpu.0.gpu‚75.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚269.233' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.74433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/kÀt…¨6‚(:ú ÁÎöÒÛξË) disk./.usagePercent‚76.22112478539796 disk.dm-1.in‚0# disk.dm-1.out‚15590.72998046875 disk.rbd0.out‚0' disk.nvme1c1n1.out‚48272.6123046875 cpu‚25.703442948033697 network.sent‚ 2.7876905e+07 network.recv‚ 4.091974e+06) disk.nvme0c0n1.out‚103385.83154296875' disk.nvme0n1.out‚103385.83154296875 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚48272.6123046875 disk.md0.in‚0# proc.memory.rssMB‚ 3025.80078125% disk./.usageGB‚1170.1848182678223! disk.dm-0.out‚134588.33203125 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 707.40234375$ memory_percent‚97.06113338470459* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375( proc.memory.availableMB‚ 481.50390625W(º‚ž©6‚(:“ ÁÎöÒà±ýË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚120.614' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.20466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V<ä¶~ª6‚(:ó ÐÎöҤͳË# proc.memory.rssMB‚ 3025.80078125 network.sent‚ 2.7881853e+07) disk.nvme0c0n1.out‚103386.47607421875 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚103386.47607421875% disk.nvme1n1.out‚48272.8779296875$ memory_percent‚97.05963134765625" proc.memory.availableMB‚481.75 proc.cpu.threads‚17 disk.rbd0.out‚0 network.recv‚ 4.095779e+06) disk./.usagePercent‚76.22111857354767% disk./.usageGB‚1170.1847229003906 disk.md0.out‚ 707.51171875! disk.dm-0.out‚134588.91796875' disk.nvme1c1n1.out‚48272.8779296875* proc.memory.percent‚18.468022346496582 disk.md0.in‚0 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 cpu‚25.70374123426735 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15590.79638671875 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375õD{‘«6‚(:† ÐÎöÒ˜ôïË gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚277.08' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.36 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'¥Ê%4¬6/ _wandbruntime‚15756  _runtime‚15756e¦£ Ê­6Ä % _timestamp‚1.7845225831700928e+09 _runtime‚15756.745711444 _step‚1427 & lr-AdamW/pg1‚2.732229345037847e-05 & lr-AdamW/pg2‚2.732229345037847e-05 trainer/global_step‚17349f¨§Ï®6É trainer/global_step‚17349 % _timestamp‚1.7845225831700928e+09 _runtime‚15756.745711444 _step‚1427 & lr-AdamW/pg1‚2.732229345037847e-05 & lr-AdamW/pg2‚2.732229345037847e-05“ Ó·ø 4¯6/ _wandbruntime‚15756  _runtime‚15756õÐ" (°6†P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312168987 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30267501 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2660698115053680 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171675790 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30267504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2676275858857984 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2656118973896192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20126479965632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71678808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41411594 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140493199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171685933 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚15756.746845472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1428 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2676276217321472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20269838458368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2660698472654314 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140493197 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171685933 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2c¥Y}±6‹P N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20126820526906 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20157226385920 HBDeviceStatsMonitor.í†H8”%on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20269838458368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2660698472654314 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140493197 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171685933 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312168987 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30267501 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2660698115053680 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171675790 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30267504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2676275858857984 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2656118973896192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20126479965632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71678808 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41411594 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140493199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171685933 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚15756.746845472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1428 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2676276217321472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30269235 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20127928915270 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312196595 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2676425601929728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140500990 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2660847011710150 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312186436 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171685446 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71682811 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41413862 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20157995060736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2676425601929728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312196595 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71683100 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚576809997225984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536,ÙÁ¦'´6”N =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312196595 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2676425601929728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140500990 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2660847011710150 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312186436 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171685446 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71682811 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41413862 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20157995060736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2676425601929728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312196595 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71683100 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚576809997225984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20127588353884 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30269235 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20127928915270 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232• šgŸõ4µ6/ _wandbruntime‚15757  _runtime‚15757×t` ¶6‡ 4loss/FBt1/assignment_loss‚0.00041768347942126497 3loss/FBt2/assignment_loss‚6.961278209702404e-05 /loss/FRt1/detection_loss‚0.4684534851797199 epoch‚43 0loss/FRt1/assignment_loss‚1.4923358125297574 3loss/SRqqt2/assignment_loss‚0.46492305633754366 /loss/FBt2/detection_loss‚0.3228287817674711 $loss/total_loss‚6.90713567969648 0loss/FRt2/assignment_loss‚0.6519148419706189 /loss/FRt2/detection_loss‚1.5939942653907453 /loss/FBt1/detection_loss‚0.3765954330028055 _runtime‚ 15757.3764041 _step‚1430 1loss/SRqqt1/detection_loss‚0.5472591789231848 1loss/SRqqt2/detection_loss‚0.5883512738790082 trainer/global_step‚17349 $ _timestamp‚1.784522583800895e+09 2loss/SRqqt1/assignment_loss‚0.3999922544541073•s)·6Œ 1loss/SRqqt1/detection_loss‚0.5472591789231848 1loss/SRqqt2/detection_loss‚0.5883512738790082 trainer/global_step‚17349 $ _timestamp‚1.784522583800895e+09 2loss/SRqqt1/assignment_loss‚0.3999922544541073 4loss/FBt1/assignment_loss‚0.00041768347942126497 3loss/FBt2/assignment_loss‚6.961278209702404e-05 /loss/FRt1/detection_loss‚0.4684534851797199 epoch‚43 0loss/FRt1/assignment_loss‚1.4923358125297574 3loss/SRqqt2/assignment_loss‚0.46492305633754366 /loss/FBt2/detection_loss‚0.3228287817674711 $loss/total_loss‚6.90713567969648 0loss/FRt2/assignment_loss‚0.6519148419706189 /loss/FRt2/detection_loss‚1.5939942653907453 /loss/FBt1/detection_loss‚0.3765954330028055 _runtime‚ 15757.3764041 _step‚1430– ࡲƒ¸6‚(:ø ßÎöҌӕÌ% disk.nvme1n1.out‚48273.5615234375 disk.md0.in‚0 disk.md0.out‚ 707.90234375# disk.dm-0.out‚134591.1591796875 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375) disk.nvme0c0n1.out‚103388.97509765625 disk.nvme0n1.in‚ 309.37109375$ memory_percent‚97.05986976623535 network.sent‚ 2.789749e+07( disk./.usagePercent‚76.2211235430279 disk.rbd0.in‚0 disk.nvme1n1.in‚ 229.87109375' proc.memory.availableMB‚ 481.7109375# proc.memory.rssMB‚ 3025.80078125 network.recv‚ 4.098518e+06* proc.memory.percent‚18.468022346496582 cpu‚25.704051891216988 proc.cpu.threads‚17$ disk./.usageGB‚1170.184799194336 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15590.87841796875 disk.rbd0.out‚0' disk.nvme0n1.out‚103388.97509765625" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚48273.5615234375pþõî¹6‚(:„ ßÎöÒЯÔÌ gpu.0.gpu‚91.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.0' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.0 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°÷J^º6‚(:ú îÎöÒù©ªË network.recv‚ 4.103852e+06' disk.nvme0c0n1.out‚103390.478515  m/'625 disk.nvme0n1.in‚ 309.37109375 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0# proc.memory.rssMB‚ 3025.80078125 cpu‚25.704304132104422) disk./.usagePercent‚76.22112279760587" disk.nvme0c0n1.in‚ 309.37109375% disk.nvme0n1.out‚103390.478515625" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚48274.20458984375 disk.md0.out‚ 708.30078125$ disk.dm-0.out‚134592.32568359375 disk.dm-1.in‚0 network.sent‚ 2.7948269e+07 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0" disk.dm-1.out‚15590.9970703125 disk.rbd0.out‚0( proc.memory.availableMB‚ 481.71484375* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17% disk./.usageGB‚1170.1847877502441( disk.nvme1c1n1.out‚48274.20458984375$ memory_percent‚97.05984592437744í¢ ž»6‚(:“ îÎöÒ¡¨íË gpu.0.gpu‚91.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚277.929' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.64299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øGþh€¼6‚(:õ ýÎöÒª¾³Ë disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375$ disk.nvme0n1.out‚103393.86328125& disk.nvme1n1.out‚48275.27001953125$ memory_percent‚97.06156253814697* proc.memory.percent‚18.468022346496582& disk.nvme0c0n1.out‚103393.86328125 network.sent‚ 2.795326e+07 disk.md0.out‚ 708.73046875$ disk.dm-0.out‚134595.73876953125 disk.dm-1.in‚0" disk.dm-1.out‚15591.0986328125( disk.nvme1c1n1.out‚48275.27001953125# proc.memory.rssMB‚ 3025.80078125 network.recv‚ 4.107249e+06 disk.md0.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚ 229.87109375( proc.memory.availableMB‚ 481.43359375 cpu‚25.704410489581978' disk./.usagePercent‚76.221125282346% disk./.usageGB‚1170.1848258972168 disk.dm-0.in‚ 539.2421875" disk.nvme1c1n1.in‚ 229.87109375 proc.cpu.threads‚17ÞQ:Sœ½6‚(:‘ ýÎöÒÆ›¶Ë gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚254.916' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.97200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pÉÃ4¾6/ _wandbruntime‚15801  _runtime‚15801MYªàÊ¿6Ä & lr-AdamW/pg1‚2.631620680394697e-05 & lr-AdamW/pg2‚2.631620680394697e-05 trainer/global_step‚17399 % _timestamp‚1.7845226283140407e+09 _runtime‚15801.889776472 _step‚1431óŸO=ÏÀ6É & lr-AdamW/pg1‚2.631620680394697e-05 & lr-AdamW/pg2‚2.631620680394697e-05 trainer/global_step‚17399 % _timestamp‚1.7845226283140407e+09 _runtime‚15801.889776472 _step‚1431— vŸ¶V4Á6/ _wandbruntime‚15801  _runtime‚15801fP“' (Â6†P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140882847 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71880656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚598761580865024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2668125489241322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140882849 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2647943168461648 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172170585 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140882847 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20182303740314 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41526452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2663532716042752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20181963179040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 trainer/global_step‚17399 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20212795487744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2683745153067008 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313043289 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2683745153067008 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172160442 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20212454063616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚1432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2663532699003392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41526155 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20325640941568 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2647943185501008 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140882849 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2663532699003392 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30354201 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313053434 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20212795487744 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2663532716042752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚598761452789248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71880356 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30354204 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140882847 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20182303740314 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41526452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2663532716042752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20181963179040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 trainer/global_step‚17399 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20212795487744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2683745153067008 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313043289 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2683745153067008 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172160442 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20212454063616 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚1432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2663532699003392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41526155 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20325640941568 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2647943185501008 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140882849 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2663532699003392 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30354201 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313053434 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20212795487744 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2663532716042752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚598761452789248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71880356 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30354204 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313060738 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2648090616168480 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20183412128678 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30355938 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2683894896138752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41528485 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41528771 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140890642 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚17399 % _timestamp‚1.7845226289476407e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2668274028297158 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2663680990544896 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2663680973505536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172180255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313070897 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140890642 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2668273670696412 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚598795876304896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172180255 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20213905593856 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20213564162560 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20326760312832 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2663680973505536 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71884420 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚1433 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20213564162560 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172180255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313070897 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140890642 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2668273670696412 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚598795876304896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172180255 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20213905593856 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20213564162560 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20326760312832 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2663680973505536 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71884420 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚1433 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20213564162560 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313060738 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2648090616168480 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20183412128678 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30355938 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2683894896138752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41528485 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41528771 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140890642 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚17399 % _timestamp‚1.7845226289476407e+09™ †o´e4Ç6/ _wandbruntime‚15802  _runtime‚15802n›Â| È6… 0loss/FRt1/detection_loss‚0.48917770119910386 /loss/FRt2/detection_loss‚1.5379071764188348 %loss/total_loss‚7.123281039384775 # _timestamp‚1.78452262894878e+09 _step‚1434 .loss/FBt1/detection_loss‚0.333080508567813 trainer/global_step‚17399 _runtime‚15802.52424625 2loss/SRqqt1/assignment_loss‚0.3802550135493393 3loss/FBt1/assignment_loss‚0.0037861166746618225 2loss/SRqqt2/assignment_loss‚0.5010465101073858 1loss/SRqqt1/detection_loss‚0.6444299899681629 1loss/SRqqt2/detection_loss‚0.6206791999712122 0loss/FBt2/detection_loss‚0.40914494898946335 epoch‚43 0loss/FRt1/assignment_loss‚1.5244602065649815 /loss/FRt2/assignment_loss‚0.632834493494653 2loss/FBt2/assignment_loss‚0.046479173879162755•œhÉ6Š /loss/FRt2/assignment_loss‚0.632834493494653 2loss/FBt2/assignment_loss‚0.046479173879162755 0loss/FRt1/detection_loss‚0.48917770119910386 /loss/FRt2/detection_loss‚1.5379071764188348 %loss/total_loss‚7.123281039384775 # _timestamp‚1.78452262894878e+09 _step‚1434 .loss/FBt1/detection_loss‚0.333080508567813 trainer/global_step‚17399 _runtime‚15802.52424625 2loss/SRqqt1/assignment_loss‚0.3802550135493393 3loss/FBt1/assignment_loss‚0.0037861166746618225 2loss/SRqqt2/assignment_loss‚0.5010465101073858 1loss/SRqqt1/detection_loss‚0.6444299899681629 1loss/SRqqt2/detection_loss‚0.6206791999712122 0loss/FBt2/detection_loss‚0.40914494898946335 epoch‚43 0loss/FRt1/assignment_loss‚1.5244602065649815š zPBô}Ê6‚(:ò ŒÏöÒ®”†Ì& disk.nvme1n1.out‚48275.78759765625 cpu‚25.704483087954298 proc.cpu.threads‚17 network.sent‚ 2.7969028e+07 network.recv‚ 4.110676e+06 disk.md0.in‚0 disk.md0.out‚ 709.046875 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375$ disk./.usageGB‚1170.184902191162$ disk.dm-0.out‚134596.91748046875 disk.dm-1.in‚0$ disk.nvme0n1.out‚103395.30078125( proc.memory.availableMB‚ 481.65234375# proc.memory.rssMB‚ 3025.80078125 disk.dm-0.in‚ 539.2421875 disk.rbd0.out‚0& disk.nvme0c0n1.out‚103395.30078125 disk.nvme1n1.in‚ 229.87109375" memory_percent‚97.060227394104* proc.memory.percent‚18.468022346496582) disk./.usagePercent‚76.22113025182622! disk.dm-1.out‚15591.193359375 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚48275.78759765625ÞbMœË6‚(:‘ ŒÏöÒ´—ÅÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚119.921' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.97366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µß5l|Ì6‚(:ñ ›ÏöÒˆÃ¶Ë cpu‚25.704291583900527 proc.cpu.threads‚17 network.sent‚ 2.8018697e+07 disk.rbd0.out‚0' disk.nvme0c0n1.out‚103397.427734375% disk.nvme0n1.out‚103397.427734375' disk.nvme1c1n1.out‚48276.5849609375% disk.nvme1n1.out‚48276.5849609375# proc.memory.rssMB‚ 3025.80078125) disk./.usagePercent‚76.22115435380536" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0! disk.dm-1.out‚15591.244140625% disk./.usageGB‚1170.1852722167969 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚709.375# disk.dm-0.out‚134599.0556640625 disk.dm-1.in‚0$ memory_percent‚97.06175327301025( proc.memory.availableMB‚ 481.40234375* proc.memory.percent‚18.468022346496582 network.recv‚ 4.115873e+06 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375 disk.dm-0.in‚ 539.2421875 ÜŸžÍ6‚(:“ ›ÏöÒ˜í÷Ë gpu.0.gpu‚91.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚274.751' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.58366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Dµi‚Î6‚(:÷ ªÏöÒØȸË disk.dm-1.in‚0 disk.rbd0.in‚0) disk.nvme0c0n1.out‚103398.46337890625 network.sent‚ 2.8023539e+07) disk./.usagePercent‚76.10702626461743" disk.nvme1c1n1.in‚ 229.87109375# disk.dm-1.out‚15591.33837890625 disk.rbd0.out‚0( proc.memory.availableMB‚ 481.40234375# proc.memory.rssMB‚ 3025.80078125 cpu‚25.704066734745563 network.recv‚ 4.119399e+06 disk.md0.out‚ 709.55859375# disk.dm-0.out‚134599.9619140625 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚103398.46337890625$ memory_percent‚97.06175327301025* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17 disk.md0.in‚0" disk.nvme0c0n1.in‚ 309.37109375$ disk./.usageGB‚1168.433120727539% disk.nvme1c1n1.out‚48276.97265625 disk.nvme1n1.in‚ 229.87109375# disk.nvme1n1.out‚48276.97265625 disk.dm-0.in‚ 539.2421875OLr¿“Ï6‚(:ˆ ªÏöÒšœ×Ë gpu.0.gpu‚32.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.312' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸û%¤4Ð6/ _wandbruntime‚15845  _runtime‚15845;’ÎÁÌÑ6Æ ' lr-AdamW/pg2‚2.5328163670378312e-05 trainer/global_step‚17449 % _timestamp‚1.7845226717685301e+09 _runtime‚15845.344145154 _step‚1435 ' lr-AdamW/pg1‚2.5328163670378312e-05¹L/’ÑÒ6Ë _runtime‚15845.344145154 _step‚1435 ' lr-AdamW/pg1‚2.5328163670378312e-05 ' lr-AdamW/pg2‚2.5328163670378312e-05 trainer/global_step‚17449 % _timestamp‚1.7845226717685301e+09› Lé!4Ó6/ _wandbruntime‚15845  _runtime‚15845±q7Þ (Ô6†P [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚15845.348081481 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20268364589568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2691214447276032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20268023165440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20268364589568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72081946 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30440904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2675552148227696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172655237 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2691214805739520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313917591 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2670946441149952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20268023165440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141272497 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20237446392448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30440901 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚600476387480064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 134367232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 115755008 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _step‚1436 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172645094 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172645094 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2670946424110592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41641045 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2675552505828330 % _timestamp‚1.7845226717695801e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚580094843963904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580094825351680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141272499 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141272499 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2670946424110592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41641346 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2691214447276032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172655237 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20381543516160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 I¡I)Õ6‹P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2670946441149952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20268023165440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141272497 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20237446392448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30440901 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚600476387480064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 134367232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 115755008 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _step‚1436 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172645094 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172645094 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2670946424110592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41641045 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2675552505828330 % _timestamp‚1.7845226717695801e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚301 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚580094843963904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580094825351680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141272499 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141272499 Jèbatch_start/requested_bytes.large_pool.freed‚2655314701835248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2670946441149952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2670946424110592 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41641346 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2691214447276032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172655237 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20381543516160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚15845.348081481 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20268364589568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2691214447276032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20268023165440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20268364589568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72081946 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30440904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2675552148227696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172655237 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2691214805739520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313917591 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180œ Ø0u4Ö6/ _wandbruntime‚15845  _runtime‚15845…F½Ð'×6N MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30442635 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2655462149542080 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30442638 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2671094715652096 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2671094698612736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2671094715652096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚17449 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313945199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41643618 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313945199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20269133264384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2655462132502720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141280292 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313935040 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141280292 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2691364190347776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20269474695680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚600510553310720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141280290 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313935040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2675700687283420 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72085974 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41643339 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172664907 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172654750 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚600510681379328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172664907 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20238895342086 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580128005621760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172654750 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20269474695680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚15845.960311079 _step‚1437 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20269133264384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20382657145344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141280290 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚580128024233984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2675701044884166 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2691363831877120 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2671094698612736eËXÆ'Ø6•N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2691363831877120 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2671094698612736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30442635 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2655462149542080 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30442638 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2671094715652096 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2671094698612736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2671094715652096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 trainer/global_step‚17449 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313945199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41643618 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313945199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20269133264384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2655462132502720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141280292 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313935040 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141280292 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2691364190347776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20269474695680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚600510553310720 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141280290 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313935040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2675700687283420 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72085974 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41643339 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172664907 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172654750 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚600510681379328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172664907 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20238895342086 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580128005621760 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172654750 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20269474695680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚15845.960311079 _step‚1437 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20269133264384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20382657145344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141280290 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚580128024233984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2675701044884166 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ö‹Y‰4Ù6/ _wandbruntime‚15845  _runtime‚15845ÚëËêÚ6 /loss/FRt2/detection_loss‚1.3586043598725113 /loss/FBt2/detection_loss‚0.3232393009393305 %loss/total_loss‚6.668422555751954 epoch‚43 trainer/global_step‚17449 % _timestamp‚1.7845226723858497e+09 0loss/FRt1/assignment_loss‚1.5074801264126176 2loss/SRqqt1/assignment_loss‚0.4009130915421944 4loss/FBt1/assignment_loss‚0.00014793931491407567 0loss/FRt1/detection_loss‚0.47614326891314473 4loss/FBt2/assignment_loss‚2.4275646716646128e-05 _runtime‚15845.96112779 _step‚1438 0loss/FRt2/assignment_loss‚0.6346158940362704 1loss/SRqqt1/detection_loss‚0.6240173446660241 1loss/SRqqt2/detection_loss‚0.5778550281308984 0loss/FBt1/detection_loss‚0.35343427896550195 3loss/SRqqt2/assignment_loss‚0.41194764731182953m‘>•Û6’ trainer/global_step‚17449 % _timestamp‚1.7845226723858497e+09 0loss/FRt1/assignment_loss‚1.5074801264126176 2loss/SRqqt1/assignment_loss‚0.4009130915421944 4loss/FBt1/assignment_loss‚0.00014793931491407567 0loss/FRt1/detection_loss‚0.47614326891314473 4loss/FBt2/assignment_loss‚2.4275646716646128e-05 _runtime‚15845.96112779 _step‚1438 0loss/FRt2/assignment_loss‚0.6346158940362704 1loss/SRqqt1/detection_loss‚0.6240173446660241 1loss/SRqqt2/detection_loss‚0.5778550281308984 0loss/FBt1/detection_loss‚0.35343427896550195 3loss/SRqqt2/assignment_loss‚0.41194764731182953 /loss/FRt2/detection_loss‚1.3586043598725113 /loss/FBt2/detection_loss‚0.3232393009393305 %loss/total_loss‚6.668422555751954 epoch‚43ž ”«©VƒÜ6‚(:ø ¹ÏöÒµ—ÍË network.sent‚ 2.8028423e+07 disk.dm-0.in‚ 539.2421875" disk.dm-0.out‚134601.287109375 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme1c1n1.out‚48277.455078125 network.recv‚ 4.122759e+06% disk./.usageGB‚1168.4331398010254 disk.md0.in‚0" disk.nvme1c1n1.in‚ 229.87109375# memory_percent‚97.0526933670044( proc.memory.availableMB‚ 482.88671875) disk.nvme0c0n1.out‚103399.91943359375) disk./.usagePercent‚76.10702750698749# disk.dm-1.out‚15591.41650390625 disk.rbd0.in‚0$ disk.nvme1n1.out‚48277.455078125# proc.memory.rssMB‚ 3025.80078125 cpu‚25.703873325468585 proc.cpu.threads‚17 disk.md0.out‚ 709.80078125 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚103399.91943359375 disk.nvme1n1.in‚ 229.87109375* proc.memory.percent‚18.468022346496582" disk.nvme0c0n1.in‚ 309.37109375'a¶SžÝ6‚(:“ ¹ÏöÒ»ÜÄÌ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚275.315' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.77166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡Z*ò~Þ6‚(:ó ÈÏöÒ«É²Ë cpu‚25.703602857490345) disk./.usagePercent‚76.10699744163206" disk.dm-0.out‚134601.880859375 disk.dm-1.out‚ 15591.5078125 disk.rbd0.in‚0& disk.nvme0n1.out‚103400.7294921875# memory_percent‚97.0527172088623 proc.cpu.threads‚17 network.sent‚ 2.8092836e+07% disk./.usageGB‚1168.4326782226562 disk.dm-0.in‚ 539.2421875 network.recv‚ 4.128449e+06% disk.nvme1n1.out‚48277.9267578125 disk.md0.out‚ 710.08203125 disk.dm-1.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3025.80078125 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0( disk.nvme0c0n1.out‚103400.7294921875 disk.nvme0n1.in‚ 309.37109375' disk.nvme1c1n1.out‚48277.9267578125' proc.memory.availableMB‚ 482.8828125* proc.memory.percent‚18.468022346496582c¡‘ùžß6‚(:“ ÈÏöÒ®††Ì gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚273.809' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.26966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÓuà6‚(:ö ×ÏöÒ‘Ò°Ë% disk./.usageGB‚1168.4332847595215 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚48278.81103515625 disk.md0.out‚ 710.703125 disk.dm-0.in‚ 539.2421875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0n1.out‚103403.8701171875" disk.nvme1c1n1.in‚ 229.87109375' proc.memory.availableMB‚ 482.8828125# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17( disk.nvme0c0n1.out‚103403.8701171875( disk.nvme1c1n1.out‚48278.81103515625 cpu‚25.703379384429134) disk./.usagePercent‚76.10703694899993 disk.md0.in‚0$ disk.dm-0.out‚134604.53955078125 disk.dm-1.in‚0 disk.dm-1.out‚15591.56640625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375# memory_percent‚97.0527172088623 network.sent‚ 2.8097722e+07 network.recv‚ 4.131816e+06j~ÖKžá6‚(:“ ×ÏöÒ÷ýõË gpu.0.gpu‚83.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚270.184' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.06133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00J4â6/ _wandbruntime‚15888  _runtime‚15888ƒæŽÊã6Ä _step‚1439 & lr-AdamW/pg1‚2.435822750983671e-05 & lr-AdamW/pg2‚2.435822750983671e-05 trainer/global_step‚17499 % _timestamp‚1.7845227145371392e+09 _runtime‚15888.112770133$IAÏä6É % _timestamp‚1.7845227145371392e+09 _runtime‚15888.112770133 _step‚1439 & lr-AdamW/pg1‚2.435822750983671e-05 & lr-AdamW/pg2‚2.435822750983671e-05 trainer/global_step‚17499Ÿ À)Q4å6/ _wandbruntime‚15888  _runtime‚15888±NžÈyæ6†P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2678360149217792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30527604 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173129746 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2678360149217792 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2662686235208848 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173139889 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.s„Ôþ…“!mall_pool.allocated‚20323933691392 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141662149 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2662686252248208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41755778 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2682979522415338 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2698683741485056 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2678360166257152 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2678360166257152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72283650 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚602191086191104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20292929605856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2682979164814704 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 314791893 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚581753857467904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚602191207975424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 314802038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚17499 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173139889 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72283379 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20437350507520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20293270167130 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173129746 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚15888.113562315 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20323592267264 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2698683741485056 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141662149 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799³C¬Ó(ç6‹P ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚602191086191104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20292929605856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2682979164814704 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 314791893 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚581753857467904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚602191207975424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 314802038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚17499 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 173139889 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72283379 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20437350507520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20293270167130 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173129746 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _runtime‚15888.113562315 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20323592267264 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2698683741485056 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141662149 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2678360149217792 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30527604 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173129746 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2678360149217792 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2662686235208848 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173139889 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20323933691392 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141662149 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2662686252248208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41755778 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2682979522415338 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2698683741485056 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2678360166257152 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2678360166257152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72283650 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0  ŠrLâ4è6/ _wandbruntime‚15888  _runtime‚15888P{^ø'é6N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173149559 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141669940 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 314819501 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141669940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41758050 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚581787037737984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2698833126086144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173139402 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2698833484556800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20325043797504 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20324702366208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20438366866432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2683127703870428 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20294378555494 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141669942 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72287385 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2698833484556800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚602225385992192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1441 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30529338 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚602225503575040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2678508423719936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚581787019125760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 314809342 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 314819501 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2678508423719936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 314809342 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30529335 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173139402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20324702366208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2683128061471174 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173149559 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2678508440759296 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 % _timestamp‚1.7845227151554816e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2678508440759296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20438465837056 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20325043797504 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2662833682915680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344;˜3„ðê6”N OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72287647 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41758309 _runtime‚15888.731360088 trainer/global_step‚17499 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 314819501 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141669940 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41758050 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚581787037737984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2698833126086144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173139402 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2698833484556800 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20325043797504 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20324702366208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20438366866432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2683127703870428 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20294378555494 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141669942 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72287385 @:DeviceStatsMonitor.on_train_batch_end/num_syn=Ôîª*c_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2698833484556800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚602225385992192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1441 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30529338 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚602225503575040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2678508423719936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚581787019125760 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 314809342 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 314819501 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2678508423719936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 314809342 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30529335 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173139402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20324702366208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2683128061471174 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173149559 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2678508440759296 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 % _timestamp‚1.7845227151554816e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2678508440759296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20438465837056 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20325043797504 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2662833682915680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173149559 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141669940 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚583412870971904 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20493158860800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142051799 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142051797 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2706153035694080 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20379161369088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2685773874324992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20379502793216 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 1736143989í~Á'ù6ŠP GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20493158860800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142051799 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142051797 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2706153035694080 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20379161369088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2685773874324992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20379502793216 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173614398 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2685773874324992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72484881 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚583412870971904 J‚4ú6/ _wandbruntime‚15931  _runtime‚15931Õƒ‚'û6N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2690555078058182 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2685922148827136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2690554720457436 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315693803 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315683644 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142059590 _step‚1445 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2685922165866496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41872834 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142059592 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173624054 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2706302420295168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2670205216289280 _runtime‚15931.849483552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20380612899328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20494275111424 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2670205199249920 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173634211 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142059592 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315693803 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41873122 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142059590 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚603940206673408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚583446032629760 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173634211 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72489160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30616038 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20380612899328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20380271468032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72488869 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2685922148827136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2690554720457436 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315693803 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315683644 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142059590 _step‚1445 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2685922165866496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41872834 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142059592 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173624054 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2706302420295168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2670205216289280 _runtime‚15931.849483552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20380612899328 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20494275111424 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2670205199249920 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173634211 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142059592 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315693803 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41873122 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142059590 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚603940206673408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚583446032629760 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173634211 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72489160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30616038 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20380612899328 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20380271468032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72488869 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2690555078058182 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FÉ…7à & lr-AdamW/pg1‚2.247292412960461e-05 & lr-AdamW/pg2‚2.247292412960461e-05 trainer/global_step‚17599 % _timestamp‚1.7845228011469843e+09 _runtime‚15974.72263804 _step‚1447`ÛgXΆ7È & lr-AdamW/pg1‚2.247292412960461e-05 & lr-AdamW/pg2‚2.247292412960461e-05 trainer/global_step‚17599 % _timestamp‚1.7845228011469843e+09 _runtime‚15974.72263804 _step‚1447§ ù"4‡7/ _wandbruntime‚15974  _runtime‚15974fïVˆ7…P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20434730470912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316550642 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41985925 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2693187616471552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2713622688366592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚605620871313408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2693187599432192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2713622329903104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174099050 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚585071865863680 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2677429301956048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142441447 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚585071884475904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2713622329903104 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72686929 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2697833555589354 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142441449 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2693187599432192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 % _timestamp‚1.7845228011490912e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20403896032672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20434730470912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30701001 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20548986837504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72686653 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142441447 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174099050 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20404236593946 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 316540497 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2677429318995408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 239956131848 (‰7ŠP % _timestamp‚1.7845228011490912e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20403896032672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20434730470912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30701001 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20548986837504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72686653 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142441447 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174099050 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20404236593946 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 316540497 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2677429318995408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20434730470912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 316550642 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41985925 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2693187616471552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2713622688366592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚605620871313408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2693187599432192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2713622329903104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174099050 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚585071865863680 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2677429301956048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142441447 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚585071884475904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2713622329903104 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72686929 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2697833555589354 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142441449 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2693187599432192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452¨ àö(êyŠ7‚(:î ±ÐöÒ²‡³Ë disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 network.sent‚ 2.8241512e+07 disk.dm-0.in‚ 539.2421875" disk.dm-1.out‚15592.2705078125 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚48283.07080078125# proc.memory.rssMB‚ 3025.80078125 cpu‚25.702391024781083$ disk./.usageGB‚1168.432502746582# disk.dm-0.out‚134612.7880859375' disk.nvme0n1.out‚103413.04931640625* proc.memory.percent‚18.468022346496582 network.recv‚ 4.155747e+06 disk.rbd0.in‚0 disk.md0.out‚712.75# memory_percent‚97.0611572265625! proc.memory.availableMB‚481.5 proc.cpu.threads‚17) disk./.usagePercent‚76.10698601182753 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375) disk.nvme0c0n1.out‚103413.04931640625 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚48283.07080078125AÏ~“‹7‚(:ˆ ±ÐöÒ¥¦ªË gpu.0.gpu‚50.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚174.237' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚58.079 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-Û“4Œ7/ _wandbruntime‚15975  _runtime‚15975Pc'7N E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 316568105 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2693335873934336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142449240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845228017728794e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142449242 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41987865 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚605655169088000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20550104342016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2677576749662880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72690600 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20436182001152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20405344982310 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142449240 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2713771714504192 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2713772072974848 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174118863 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20435840569856 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72690897 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚605655036825088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _step‚1449 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20549990691328 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142449242 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316557946 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2697982094645190 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚15975.348764244 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316568105 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174108706 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚585105046133760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2713772072974848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174108706 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174118863 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316557946 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚17599 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30702735 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20435840569856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2677576732623520 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2693335890973696 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2693335890973696 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 316568105 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2693335873934336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142449240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845228017728794e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142449242 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41987865 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚605655169088000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20550104342016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2677576749662880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72690600 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20436182001152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20405344982310 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142449240 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2713771714504192 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2713772072974848 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174118863 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20435840569856 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72690897 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚605655036825088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _step‚1449 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20549990691328 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142449242 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316557946 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2697982094645190 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚15975.348764244 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316568105 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174108706 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚585105046133760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2713772072974848 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K=DeviceStatsMonitor.o'kiùgn_train_batch_end/active.small_pool.freed‚ 174108706 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174118863 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316557946 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 trainer/global_step‚17599 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30702735 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20435840569856 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2677576732623520 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29370933771133423 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7557252049446106 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24397854506969452 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34174826741218567 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22972401976585388 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2777383029460907 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6357162594795227 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7149215340614319 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5753898620605469 7validation_average_jet_accuracy‚0.37118783593177795 epoch‚43 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5812592506408691 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08199725300073624 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6382380723953247 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2983958423137665 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8812333941459656 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2348446547985077 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3475225865840912 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5948658585548401 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 )loss/val_total_loss‚7.423175879081072 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06096220761537552 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8246976733207703 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03982879966497421 particle/accuracy_1_of_2‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07691577076911926 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9971181154251099 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7"Purity/1FRt*SRqqt1FBt/event_purity‚0.50429368019104 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6294993758201599 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.29158058762550354 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2719017565250397 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 jet/accuracy_1_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 *jet/accuracy_1_of_1‚0.3301502466201782 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08188480138778687 particle/accuracy_4_of_5‚1 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6891400218009949 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7591648101806641 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6565124988555908 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6198133230209351 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.45940032601356506 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.833541214466095 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.615394115447998 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_2_of_4‚0.7386557459831238 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6203085780143738 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6628209948539734 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7347881197929382 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8403250575065613 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05063994601368904 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11847344785928726 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8267195820808411 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8705515265464783 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6446986794471741 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6196291446685791 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45498785376548767 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5934517979621887 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9903878569602966 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08272504806518555 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_4‚0.9672623872756958 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5722206234931946 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4732554256916046 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7711263298988342 *jet/accuracy_1_of_2‚0.7702288627624512 _step‚1451 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7008329033851624 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05076179653406143 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5958333015441895 (particle/accuracy‚0.8531241416931152 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34655308723449707 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9184459447860718 particle/accuracy_2_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07691577076911926 % _timestamp‚1.7845228189508307e+09 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20603062212467194 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6182546019554138 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30295976996421814 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6673916578292847 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5626014471054077 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6995818018913269 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 _runtime‚15992.52701674 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30241283774375916 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7302227020263672 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6846699714660645 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25596415996551514 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09184560924768448 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.6448534727096558 jet/accuracy_4_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9428967237472534 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22001640498638153 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2832269072532654 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2913457453250885 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.29158058762550354 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3148537278175354 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09836918115615845 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 /particle/accuracy_4_of_4‚0.9539412260055542 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3389335870742798 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23187464475631714 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6777058839797974 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26713433861732483 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.593407928943634 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7711263298988342 /particle/accuracy_3_of_3‚0.9994120597839355 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6678707003593445 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9907387495040894 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08145104348659515 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.833541214466095 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2879144549369812 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6188273429870605 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2227555513381958 +jet/accuracy_3_of_3‚0.07427539676427841 particle/accuracy_3_of_4‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27705731987953186 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 trainer/global_step‚17599 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 particle/accuracy_1_of_4‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48415929079055786 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6438037157058716 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5829368829727173 +jet/accuracy_2_of_2‚0.16036783158779144 particle/accuracy_1_of_3‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7120087146759033 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *particle/specificity‚0.910085141658783 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2918108403682709 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 *jet/accuracy_2_of_3‚0.5613309144973755 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4845188558101654 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9097657799720764 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2099512219429016 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30231142044067383 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2658953368663788 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5733700394630432 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 'particle/f_score‚0.6534599661827087 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.645117998123169 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5922372937202454 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7096226215362549 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3034698963165283 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5803303718566895 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293$-÷i`1–7Ëz 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 _runtime‚15992.52701674 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30241283774375916 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7302227020263672 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6846699714660645 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25596415996551514 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09184560924768448 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.6448534727096558 jet/accuracy_4_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9428967237472534 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22001640498638153 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2832269072532654 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2913457453250885 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.29158058762550354 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3148537278175354 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09836918115615845 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 /particle/accuracy_4_of_4‚0.9539412260055542 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3389335870742798 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23187464475631714 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6777058839797974 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26713433861732483 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.593407928943634 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7711263298988342 /particle/accuracy_3_of_3‚0.9994120597839355 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6678707003593445 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9907387495040894 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08145104348659515 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.833541214466095 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2879144549369812 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6188273429870605 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2227555513381958 +jet/accuracy_3_of_3‚0.07427539676427841 particle/accuracy_3_of_4‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27705731987953186 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 trainer/global_step‚17599 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 particle/accuracy_1_of_4‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48415929079055786 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6438037157058716 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5829368829727173 +jet/accuracy_2_of_2‚0.16036783158779144 particle/accuracy_1_of_3‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7120087146759033 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *particle/specificity‚0.910085141658783 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2918108403682709 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 *jet/accuracy_2_of_3‚0.5613309144973755 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4845188558101654 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46759259700775146 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9097657799720764 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2099512219429016 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30231142044067383 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2658953368663788 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5733700394630432 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 'particle/f_score‚0.6534599661827087 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.645117998123169 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5922372937202454 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7096226215362549 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 jet/accuracy_2_of_5‚1 particle/accuracy_2_of_5‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3034698963165283 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5803303718566895 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974238276481628 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5733700394630432 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29370933771133423 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7557252049446106 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24397854506969452 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34174826741218567 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22972401976585388 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2777383029460907 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6357162594795227 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7149215340614319 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5753898620605469 7validation_average_jet_accuracy‚0.37118783593177795 epoch‚43 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5812592506408691 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08199725300073624 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6382380723953247 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2983958423137665 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8812333941459656 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2348446547985077 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3475225865840912 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5948658585548401 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 )loss/val_total_loss‚7.423175879081072 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06096220761537552 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8246976733207703 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03982879966497421 particle/accuracy_1_of_2‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07691577076911926 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9971181154251099 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7"Purity/1FRt*SRqqt1FBt/event_purity‚0.50429368019104 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6294993758201599 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.29158058762550354 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2719017565250397 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 jet/accuracy_1_of_5‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 *jet/accuracy_1_of_1‚0.3301502466201782 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08188480138778687 particle/accuracy_4_of_5‚1 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6891400218009949 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7591648101806641 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6565124988555908 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6198133230209351 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.45940032601356506 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.833541214466095 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.615394115447998 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_2_of_4‚0.7386557459831238 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6203085780143738 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6628209948539734 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7347881197929382 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8403250575065613 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05063994601368904 :"Purity/Å3F(ñ 2FRt0SRqqt*FBt/event_purity‚0.11847344785928726 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8267195820808411 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8705515265464783 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6446986794471741 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6196291446685791 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45498785376548767 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5934517979621887 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9903878569602966 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08272504806518555 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_4‚0.9672623872756958 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5722206234931946 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4732554256916046 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7711263298988342 *jet/accuracy_1_of_2‚0.7702288627624512 _step‚1451 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7008329033851624 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05076179653406143 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5958333015441895 (particle/accuracy‚0.8531241416931152 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34655308723449707 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9184459447860718 particle/accuracy_2_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07691577076911926 % _timestamp‚1.7845228189508307e+09 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20603062212467194 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6182546019554138 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30295976996421814 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6673916578292847 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5626014471054077 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6995818018913269 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129« WM}—7‚(:ò ÏÐöÒßØþË network.recv‚ 4.16543e+06 disk.rbd0.out‚0% disk.nvme0n1.out‚103416.126953125( disk.nvme1c1n1.out‚48284.11572265625& disk.nvme1n1.out‚48284.11572265625 disk.md0.out‚ 713.2578125$ memory_percent‚97.06089496612549( proc.memory.availableMB‚ 481.54296875 disk.dm-1.out‚ 15592.4765625 proc.cpu.threads‚17 network.sent‚ 2.8357238e+07) disk./.usagePercent‚76.10699346604788" disk./.usageGB‚1168.4326171875 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0c0n1.out‚103416.126953125 disk.nvme0n1.in‚ 309.37109375 disk.md0.in‚0$ disk.dm-0.out‚134615.58154296875# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 cpu‚25.705302568975185D‰Šž˜7‚(:“ ÏÐöÒÝï¿Ì gpu.0.gpu‚55.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚290.134' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.71133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+3ò$ƒ™7‚(:ø ÞÐöÒúПË" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 713.63671875# proc.memory.rssMB‚ 3025.80078125 cpu‚25.705409221058517# disk.dm-1.out‚15592.59814453125% disk.nvme0n1.out‚103417.373046875% disk.nvme1n1.out‚48284.7373046875* proc.memory.percent‚18.468022346496582 network.sent‚ 2.8401368e+07( disk./.usagePercent‚76.1070128470208$ disk.dm-0.out‚134616.50732421875 disk.dm-1.in‚0' disk.nvme0c0n1.out‚103417.373046875' disk.nvme1c1n1.out‚48284.7373046875 disk.md0.in‚0( proc.memory.availableMB‚ 481.54296875 network.recv‚ 4.170568e+06% disk./.usageGB‚1168.4329147338867 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375$ memory_percent‚97.06089496612549 proc.cpu.threads‚17 disk.dm-0.in‚ 539.2421875" disk.nvme0c0n1.in‚ 309.37109375Ry…š7‚(:’ ÞÐöÒ¢©òË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚119.629' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.876333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ|….›7‚(:ô íÐöÒõÕ–Ë% disk./.usageGB‚1168.4329719543457 disk.dm-0.out‚ 134618.34375 disk.rbd0.in‚0) disk.nvme0c0n1.out‚103419.36865234375 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚103419.36865234375" disk.nvme1c1n1.in‚ 229.87109375) disk./.usagePercent‚76.10701657413097 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚48285.458984375 disk.md0.out‚ 714.0078125# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 network.sent‚ 2.8407375e+07 disk.rbd0.out‚0& disk.nvme1c1n1.out‚48285.458984375$ memory_percent‚97.06089496612549( proc.memory.availableMB‚ 481.54296875 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚15592.66845703125 disk.md0.in‚0 cpu‚25.705419982452415 proc.cpu.threads‚17 network.recv‚ 4.173945e+06qvDžœ7‚(:“ íÐöÒ¾ËíË gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚280.693' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.56433333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úë8ª47/ _wandbruntime‚16035  _runtime‚16035€pÉž7à & lr-AdamW/pg2‚2.155767799942731e-05 trainer/global_step‚17649 $ _timestamp‚1.784522861593363e+09 _runtime‚16035.168998658 _step‚1452 & lr-AdamW/pg1‚2.155767799942731e-05ìÒΟ7È & lr-AdamW/pg1‚2.155767799942731e-05 & lr-AdamW/pg2‚2.155767799942731e-05 trainer/global_step‚17649 $ _timestamp‚1.784522861593363e+09 _runtime‚16035.168998658 _step‚1452¬ w[A'4 7/ _wandbruntime‚16035  _runtime‚16035Jr{Õ (¡7„P YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20510958832640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174675339 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42139160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30791820 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚607379091172352 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2686722091337872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20625521843712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2686722108377232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20480023656251 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2702543685454336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142953907 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142953905 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111298560 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20625614530048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 317629246 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174675339 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586753458030080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586753476642304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2702543668414976 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2707202132033483 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20510617408512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2723054285823488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142953905 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2702543668414976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 trainer/global_step‚17649 _step‚1453 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174665196 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92686336 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20479683094977 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 % _timestamp‚1.7845228615943158e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317619101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72930980 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42138890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚607378979873792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452퇀(¢7‰P aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2686722108377232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20480023656251 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2702543685454336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142953907 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142953905 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111298560 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20625614530048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 317629246 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174675339 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586753458030080 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586753476642304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2702543668414976 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2707202132033483 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20510617408512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2723054285823488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142953905 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2702543668414976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 trainer/global_step‚17649 _step‚1453 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174665196 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92686336 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20479683094977 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 % _timestamp‚1.7845228615943158e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317619101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72930980 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42138890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚607378979873792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20510958832640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174675339 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42139160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30791820 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚607379091172352 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2686722091337872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20625521843712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270­ ƒIs¢4£7/ _wandbruntime‚16035  _runtime‚16035 Ó‹¤7N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142961700 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20512068938752 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2723203670424576 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142961698 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174674852 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30793551 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142961700 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2686869539044704 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20480791483229 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845228622059298e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚17649 _runtime‚16035.781714178 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDevicÂ=úUeStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174685009 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20512068938752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30793554 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚586786638300160 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20626737822208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2702691942917120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚607413394734592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20511727507456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1454 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317646709 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72934678 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174685009 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20512068938752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30793554 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚586786638300160 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20626737822208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2702691942917120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚607413394734592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20511727507456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1454 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317646709 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72934678 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142961700 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20512068938752 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2723203670424576 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142961698 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174674852 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30793551 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142961700 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2686869539044704 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20480791483229 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845228622059298e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 trainer/global_step‚17649 _runtime‚16035.781714178 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2709957393522176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20535506869659 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2730523580032512 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175159991 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚588412471534080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2694093641750832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143343555 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚17699 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143343557 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20566527934464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2730523580032512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30878520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30878517 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚588412490146304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2714628791019857 _runtime‚16078.37092235 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2714629148620491 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175149848 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318493403 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2694093624711472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2714628791019857 _runtime‚16078.37092235 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚299 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2714629148620491 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175149848 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 318493403 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2694093624711472 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2709957393522176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20535506869659 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2730523580032512 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚296 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175159991 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚588412471534080 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2694093641750832 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143343555 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚17699 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143343557 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20566527934464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2730523580032512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30878520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30878517 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚588412490146304± ’ì*Y4³7/ _wandbruntime‚16078  _runtime‚16078m¨:/'´7N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318521011 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175169661 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73136405 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30880254 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2730673323104256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845229054088862e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143351348 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2710105685063680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143351350 trainer/global_step‚17699 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2710105668024320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20567638040576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143351350 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2730673323104256 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175159504 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2730672964633600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20567296609280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2710105685063680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42256151 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175169661 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318510852 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20536615258023 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚16078.98475015 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2714777330075581 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚588445651804160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318510852 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2710105668024320 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73136125 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20682541380096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2694241055378944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30880251 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2714777687676327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚609128321252864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚588445670416384 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180$ß]'µ7”N K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73136125 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20682541380096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2694241055378944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30880251 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2714777687676327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚609128321252864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚588445670416384 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318521011 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175169661 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73136405 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30880254 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2730673323104256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845229054088862e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143351348 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2710105685063680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143351350 trainer/global_step‚17699 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2710105668024320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20567638040576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143351350 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2730673323104256 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175159504 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2730672964633600 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20567296609280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2710105685063680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42256151 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175169661 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318510852 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20536615258023 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚16078.98475015 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2714777330075581 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚588445651804160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318510852 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2710105668024320 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656² ÏÝ 4¶7/ _wandbruntime‚16078  _runtime‚16078«Ú&Ø ·7‡ /loss/FRt1/assignment_loss‚1.395581635158178 1loss/SRqqt2/detection_loss‚0.6370180480805244 .loss/FBt1/detection_loss‚0.319623831696804 $ _timestamp‚1.784522905410035e+09 0loss/FRt2/assignment_loss‚0.6555680065434806 /loss/FRt2/detection_loss‚1.5479962245462586 1loss/SRqqt1/detection_loss‚0.6132282542436375 /loss/FBt2/detection_loss‚0.5236146129221501 %loss/total_loss‚7.040425088586309 epoch‚44 _step‚1459 2loss/SRqqt2/assignment_loss‚0.4560126944548517 2loss/FBt1/assignment_loss‚0.015482683820237412 4loss/FBt2/assignment_loss‚2.0106576099484148e-05 0loss/FRt1/detection_loss‚0.46095068972489844 _runtime‚16078.985334031 2loss/SRqqt1/assignment_loss‚0.4153283008191888 trainer/global_step‚17699O€Ï[¸7Œ 0loss/FRt2/assignment_loss‚0.6555680065434806 /loss/FRt2/detection_loss‚1.5479962245462586 1loss/SRqqt1/detection_loss‚0.6132282542436375 /loss/FBt2/detection_loss‚0.5236146129221501 %loss/total_loss‚7.040425088586309 epoch‚44 _step‚1459 2loss/SRqqt2/assignment_loss‚0.4560126944548517 2loss/FBt1/assignment_loss‚0.015482683820237412 4loss/FBt2/assignment_loss‚2.0106576099484148e-05 0loss/FRt1/detection_loss‚0.46095068972489844 _runtime‚16078.985334031 2loss/SRqqt1/assignment_loss‚0.4153283008191888 trainer/global_step‚17699 /loss/FRt1/assignment_loss‚1.395581635158178 1loss/SRqqt2/detection_loss‚0.6370180480805244 .loss/FBt1/detection_loss‚0.319623831696804 $ _timestamp‚1.784522905410035e+09³ %®kG…¹7‚(:ú šÑöÒ§Þ‰Ì" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 714.80859375# proc.memory.rssMB‚ 3025.80078125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚103424.7236328125 disk.nvme0n1.in‚ 309.37109375' disk.nvme1c1n1.out‚48821.7724609375% disk.nvme1n1.out‚48821.7724609375* proc.memory.percent‚18.468022346496582 cpu‚25.704640767917308 proc.cpu.threads‚17$ disk.dm-0.out‚134623.19970703125 disk.rbd0.out‚0 disk.md0.in‚0 network.sent‚ 2.8490427e+07 network.recv‚ 4.184835e+06 disk.dm-0.in‚ 539.2421875" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0n1.out‚103424.7236328125# memory_percent‚97.0628023147583( proc.memory.availableMB‚ 481.23046875) disk./.usagePercent‚76.10713633860465% disk./.usageGB‚1168.4348106384277 disk.dm-1.in‚0# disk.dm-1.out‚16127.72509765625Òõûžº7‚(:“ šÑöÒšÍÀÌ gpu.0.gpu‚61.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.922' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.30733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˧uz»7‚(:ï ©ÑöÒ›à°Ë# disk.dm-1.out‚16127.79931640625 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0c0n1.out‚103428.189453125$ disk.nvme1c1n1.out‚ 48822.5703125 disk.nvme1n1.in‚ 229.87109375" disk.nvme1n1.out‚ 48822.5703125 network.sent‚ 2.854018e+07 disk.md0.in‚0$ memory_percent‚97.06289768218994( proc.memory.availableMB‚ 481.21484375# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375 disk.md0.out‚ 715.03125 cpu‚25.704421533617005 network.recv‚ 4.190001e+06) disk./.usagePercent‚76.10713807792273% disk./.usageGB‚1168.4348373413086 disk.dm-0.in‚ 539.2421875% disk.nvme0n1.out‚103428.189453125" disk.nvme1c1n1.in‚ 229.87109375$ disk.dm-0.out‚134626.88720703125øt¸ßž¼7‚(:“ ©ÑöÒ‚»þË gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚252.592' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.19733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@+ÇÌd½7‚(:÷ ¸ÑöÒ¶ÆË disk.md0.out‚ 715.34375 network.sent‚ 2.8547173e+07" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme1n1.out‚48823.25732421875 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚134628.58056640625$ memory_percent‚97.06289768218994 cpu‚25.70415332196699$ disk./.usageGB‚1168.435459136963 disk.nvme0n1.in‚ 30e\n9.37109375" disk.nvme1c1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3025.80078125 proc.cpu.threads‚17 network.recv‚ 4.193663e+06 disk.rbd0.out‚0( disk.nvme0c0n1.out‚103429.9775390625 disk.md0.in‚0( proc.memory.availableMB‚ 481.21484375* proc.memory.percent‚18.468022346496582( disk.nvme1c1n1.out‚48823.25732421875) disk./.usagePercent‚76.10717857918664 disk.dm-1.in‚0" disk.dm-1.out‚16127.9072265625 disk.rbd0.in‚0& disk.nvme0n1.out‚103429.9775390625 disk.nvme1n1.in‚ 229.87109375n€ ž¾7‚(:“ ¸ÑöÒíψÌ gpu.0.gpu‚96.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚281.897' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.96566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«ÊO4¿7/ _wandbruntime‚16121  _runtime‚16121Á¶ÖîÊÀ7Ä ' lr-AdamW/pg2‚1.9782290777917835e-05 trainer/global_step‚17749 $ _timestamp‚1.784522947920546e+09 _runtime‚16121.49704364 _step‚1460 ' lr-AdamW/pg1‚1.9782290777917835e-05a¯­ÏÁ7É ' lr-AdamW/pg1‚1.9782290777917835e-05 ' lr-AdamW/pg2‚1.9782290777917835e-05 trainer/global_step‚17749 $ _timestamp‚1.784522947920546e+09 _runtime‚16121.49704364 _step‚1460´ Ê«c•4Â7/ _wandbruntime‚16121  _runtime‚16121ÙAò (Ã7…P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2717371135668736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2737992874241536 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175634500 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 319377850 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610808749802496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 trainer/global_step‚17749 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2737992874241536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2717371118629376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20737378422272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20622097036288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20621755612160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _runtime‚16121.49712323 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20622097036288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30965220 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30965217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42367813 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20737264764416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175634500 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175644643 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2722055807606865 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175644643 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73333030 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2701465158085072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7845229479212046e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 319367705 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143733207 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚590071503650304 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2722056165207499 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336w cO(Ä7ŠP KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30965217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42367813 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20737264764416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175634500 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175644643 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2722055807606865 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175644643 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73333030 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2701465158085072 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7845229479212046e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 319367705 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143733207 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚590071503650304 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2722056165207499 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2717371135668736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2737992874241536 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175634500 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 319377850 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610808749802496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 trainer/global_step‚17749 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2737992874241536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2717371118629376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20737378422272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20622097036288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20621755612160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _runtime‚16121.49712323 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20622097036288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1461 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30965220 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896µ ©¨_l4Å7/ _wandbruntime‚16122  _runtime‚16122gÿç'Æ7N K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73336959 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2717519393131520 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2701612605791904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319395313 % _timestamp‚1.7845229485376484e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610843175480320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 319385154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2717519393131520 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20738491559936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 319385154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2738142617313280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2717519410170880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2717519410170880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590104683920384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20738382103552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2722204704263335 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2722204346662589 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20592098471431 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143741000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Åò_5‘Ç7•N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 319395313 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175654313 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175644156 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20623207142400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_tr`üŠ$ain_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73337251 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2738142617313280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30966951 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚292 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143741000 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2738142258842624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2717519410170880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590104683920384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20738382103552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2722204704263335 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2722204346662589 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20592098471431 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143741000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73336959 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2717519393131520 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2701612605791904 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319395313 % _timestamp‚1.7845229485376484e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610843175480320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 319385154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2717519393131520 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20738491559936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 319385154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2738142617313280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2717519410170880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚289 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 EýÊ7 3loss/FBt2/assignment_loss‚0.0009633226015286311 0loss/FBt1/detection_loss‚0.37857140905105635 epoch‚44 % _timestamp‚1.7845229485391896e+09 _runtime‚16122.114544779 0loss/FRt2/assignment_loss‚0.6979913247493041 3loss/FBt1/assignment_loss‚0.0016591233991772208 _step‚1463 0loss/FRt1/assignment_loss‚1.4520011870484604 2loss/SRqqt1/assignment_loss‚0.3385248191125159 1loss/SRqqt2/detection_loss‚0.6516123446896405 /loss/FBt2/detection_loss‚0.3652157068031374 3loss/SRqqt2/assignment_loss‚0.41501527624880763 /loss/FRt1/detection_loss‚0.4256877523487652 /loss/FRt2/detection_loss‚1.5155791297477903 1loss/SRqqt1/detection_loss‚0.6354591145342958 $loss/total_loss‚6.87828051033448 trainer/global_step‚17749· EÅe©†Ë7‚(:û ÇÑöÒþóäË disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375) disk.nvme0c0n1.out‚103432.36376953125 disk.nvme0n1.in‚ 309.37109375 proc.cpu.threads‚17" disk.dm-1.out‚16127.9970703125% disk./.usageGB‚1168.4353828430176" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875' disk.nvme0n1.out‚103432.36376953125$ memory_percent‚97.06523418426514( proc.memory.availableMB‚ 480.83203125* proc.memory.percent‚18.468022346496582 network.recv‚ 4.197304e+06& disk.nvme1n1.out‚48824.21044921875" disk.dm-0.out‚134630.947265625 network.sent‚ 2.8563039e+07) disk./.usagePercent‚76.10717360970641( disk.nvme1c1n1.out‚48824.21044921875 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 715.74609375# proc.memory.rssMB‚ 3025.80078125 cpu‚25.70390083931045iìr‚žÌ7‚(:“ ÇÑöÒÓõ´Ì gpu.0.gpu‚55.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚292.943' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚97.64766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºlåÍ7‚(:ô ÖÑöÒ‘ÆäË) disk.nvme0c0n1.out‚103433.17138671875 disk.nvme0n1.in‚ 309.37109375 disk.md0.out‚ 716.0703125" disk.dm-0.out‚134631.505859375 disk.rbd0.out‚0' disk.nvme0n1.out‚103433.17138671875" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚48824.81787109375$ memory_percent‚97.06382751464844$ proc.memory.availableMB‚481.0625 disk.md0.in‚0 disk.dm-1.out‚16128.16015625" disk.nvme0c0n1.in‚ 309.37109375 proc.cpu.threads‚17) disk./.usagePercent‚76.10717385818042 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚48824.81787109375# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 cpu‚25.70366511625997 network.sent‚ 2.8614661e+07 network.recv‚ 4.202255e+06% disk./.usageGB‚1168.4353866577148 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0 disk.rbd0.in‚04M¨ÌžÎ7‚(:“ ÖÑöÒœ§¡Ì gpu.0.gpu‚91.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚280.615' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.53833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç!QÒ~Ï7‚(:ó åÑöÒªÍ­Ë network.recv‚ 4.205887e+06" disk.nvme0c0n1.in‚ 309.37109375! disk.dm-0.out‚134632.85546875 disk.rbd0.out‚0* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17$ disk./.usageGB‚1168.435489654541) disk.nvme0c0n1.out‚103434.80029296875" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚48825.53076171875 disk.md0.out‚ 716.47265625 disk.dm-1.in‚0 disk.dm-1.out‚ 16128.28125) disk./.usagePercent‚76.10718056697874 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚103434.80029296875 disk.md0.in‚0 disk.rbd0.in‚0# memory_percent‚97.0658540725708( proc.memory.availableMB‚ 480.73046875# proc.memory.rssMB‚ 3025.80078125( disk.nvme1c1n1.out‚48825.53076171875 disk.dm-0.in‚ 539.2421875 cpu‚25.70343125957407 network.sent‚ 2.8617583e+07 ]žÐ7‚(:“ åÑöÒÓ´øÌ gpu.0.gpu‚58.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚289.502' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.50066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k!À4Ñ7/ _wandbruntime‚16164  _runtime‚16164’–°JÌÒ7Æ _runtime‚16164.224928134 _step‚1464 ' lr-AdamW/pg1‚1.8922263716389376e-05 ' lr-AdamW/pg2‚1.8922263716389376e-05 trainer/global_step‚17799 % _timestamp‚1.7845229906493866e+09b¼!ÑÓ7Ë ' lr-AdamW/pg1‚1.8922263716389376e-05 ' lr-AdamW/pg2‚1.8922263716389376e-05 trainer/global_step‚17799 % _timestamp‚1.7845229906493866e+09 _runtime‚16164.224928134 _step‚1464¸ 6¬”4Ô7/ _wandbruntime‚16164  _runtime‚16164Ãí‡Ä (Õ7†P 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2708836691458672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144122857 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176129295 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚612523715800064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2724784860775936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚16164.229049199 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2745462168450560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2729482824193873 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20677666138112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2745462526914048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20677324713984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2745462168450560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚591730517154304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176119152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845229906499925e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20677324713984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144122857 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2724784843736576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144122855 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176129295 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2729483181794507 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20646132735201 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 320242007 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2745462526914048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73534961 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20793089182208 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2745462526914048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73534961 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20793089182208 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2708836691458672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144122857 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176129295 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚285 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚612523715800064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2724784860775936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚16164.229049199 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2745462168450560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2729482824193873 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20677666138112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2745462526914048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1¹ ¾44×7/ _wandbruntime‚16164  _runtime‚16164èXçd'Ø7N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20678776244224 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320269615 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176138965 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20647241123453 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚17799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2724933118238720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚612558009445888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73538644 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _step‚1466 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20678776244224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31053654 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42485287 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42484993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚612557881377280 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144130650 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2745611553051648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚591763678812160 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2708984122126144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚591763697424384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚16164.839461089 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2729631720850343 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2745611911522304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31053651 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2724933135278080 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2724933118238720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚612558009445888 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73538644 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _step‚1466 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20678776244224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31053654 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42485287 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42484993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚612557881377280 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144130650 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2745611553051648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚591763678812160 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2708984122126144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚591763697424384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚16164.839461089 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2729631720850343 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2745611911522304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31053651 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2724933135278080 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320269615 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176138965 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20647241123453 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚17799 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452º B Q¢4Ú7/ _wandbruntime‚16164  _runtime‚16164$âG“Û7‹ 4loss/FBt1/assignment_loss‚5.2911319754029944e-05 /loss/FRt1/detection_loss‚0.4758393230806918 0loss/FBt1/detection_loss‚0.39923930071058944 trainer/global_step‚17799 _step‚1467 0loss/FRt1/assignment_loss‚1.5468518050487883 2loss/SRqqt1/assignment_loss‚0.4347282015960057 1loss/SRqqt1/detection_loss‚0.6381814550754535 .loss/FBt2/detection_loss‚0.578101019971135 &loss/total_loss‚7.3367965534632695 epoch‚44 _runtime‚16164.839962479 0loss/FRt2/assignment_loss‚0.6594863022871064 2loss/SRqqt2/assignment_loss‚0.5190463175010973 1loss/SRqqt2/detection_loss‚0.6098829137374033 % _timestamp‚1.7845229912647002e+09 3loss/FBt2/assignment_loss‚0.0023851064182619348 /loss/FRt2/detection_loss‚1.4730018967169827vbHÜ7 3loss/FBt2/assignment_loss‚0.0023851064182619348 /loss/FRt2/detection_loss‚1.4730018967169827 4loss/FBt1/assignment_loss‚5.2911319754029944e-05 /loss/FRt1/detection_loss‚0.4758393230806918 0loss/FBt1/detection_loss‚0.39923930071058944 trainer/global_step‚17799 _step‚1467 0loss/FRt1/assignment_loss‚1.5468518050487883 2loss/SRqqt1/assignment_loss‚0.4347282015960057 1loss/SRqqt1/detection_loss‚0.6381814550754535 .loss/FBt2/detection_loss‚0.578101019971135 &loss/total_loss‚7.3367965534632695 epoch‚44 _runtime‚16164.839962479 0loss/FRt2/assignment_loss‚0.6594863022871064 2loss/SRqqt2/assignment_loss‚0.5190463175010973 1loss/SRqqt2/detection_loss‚0.6098829137374033 % _timestamp‚1.7845229912647002e+09» ‰ÿZ€Ý7‚(:õ ôÑöÒ‹ÂñË disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚103435.55126953125 disk.md0.out‚ 716.7109375 disk.dm-0.in‚ 539.2421875* proc.memory.percent‚18.468022346496582) disk.nvme0c0n1.out‚103435.55126953125" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚48825.99462890625$ memory_percent‚97.06602096557617# proc.memory.rssMB‚ 3025.80078125 network.recv‚ 4.209673e+06) disk./.usagePercent‚76.10718305171885! disk.dm-1.out‚16128.388671875 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-0.out‚134633.4296875& proc.memory.availableMB‚ 480.703125 network.sent‚ 2.863642e+07 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme1c1n1.out‚48825.99462890625 cpu‚25.703210088559533 proc.cpu.threads‚17% disk./.usageGB‚1168.4355278015137" disk.nvme0c0n1.in‚ 309.37109375ÇhæžÞ7‚(:“ ôÑöÒ·Ä¿Ì gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚272.252' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.75066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚—¢Mß7‚(:ö ƒÒöÒŸÓË disk.md0.in‚0 disk.md0.out‚ 717.15625 disk.dm-0.in‚ 539.2421875! disk.dm-0.out‚134635.79296875( disk./.usagePercent‚76.1071040369831' disk.nvme0n1.out‚103438.13916015625" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚48826.81298828125& disk.nvme1n1.out‚48826.81298828125 disk.dm-1.in‚0) disk.nvme0c0n1.out‚103438.13916015625$ memory_percent‚97.06661701202393 network.recv‚ 4.214629e+06 disk.nvme0n1.in‚ 309.37109375 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375( proc.memory.availableMB‚ 480.60546875* proc.memory.percent‚18.468022346496582 cpu‚25.702993249297247% disk./.usageGB‚1168.4343147277832 disk.nvme1n1.in‚ 229.87109375! disk.dm-1.out‚16128.458984375 disk.rbd0.out‚0# proc.memory.rssMB‚ 3025.80078125 proc.cpu.threads‚17 network.sent‚ 2.8684965e+07 sùBœà7‚(:‘ ƒÒöÒÉâòË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.02' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.339999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÓf‚á7‚(:÷ ’ÒöÒ™…ÕË% disk./.usageGB‚1168.4352836608887 disk.md0.in‚0 disk.md0.out‚ 717.77734375 disk.dm-0.in‚ 539.2421875! disk.dm-0.out‚134636.87109375" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.06661701202393* proc.memory.percent‚18.468022346496582 network.sent‚ 2.8691117e+07 network.recv‚ 4.218503e+06 disk.dm-1.in‚0' disk.nvme0c0n1.out‚103439.537109375% disk.nvme0n1.out‚103439.537109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚48828.10400390625# proc.memory.rssMB‚ 3025.80078125 disk.rbd0.in‚0( proc.memory.availableMB‚ 480.60546875( disk./.usagePercent‚76.1071671493821# disk.dm-1.out‚16128.74267578125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚48828.10400390625 cpu‚25.702804049955745 proc.cpu.threads‚174°x±žâ7‚(:“ ’ÒöÒ¨öºÌ gpu.0.gpu‚93.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚235.353' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚78.45100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:×.4ã7/ _wandbruntime‚16207  _runtime‚16207úÖ¼GÉä7à & lr-AdamW/pg2‚1.808075506687152e-05 trainer/global_step‚17849 % _timestamp‚1.7845230337856493e+09 _runtime‚16207.36123144 _step‚1468 & lr-AdamW/pg1‚1.808075506687152e-05E†øwÎå7È & lr-AdamW/pg1‚1.808075506687152e-05 & lr-AdamW/pg2‚1.808075506687152e-05 trainer/global_step‚17849 % _timestamp‚1.7845230337856493e+09 _runtime‚16207.36123144 _step‚1468¼ gæ W4æ7/ _wandbruntime‚16207  _runtime‚16207@^ÄÔ– ç7†P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2732198568843776 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2716208224832272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _runtime‚16207.362187241 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176603804 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2752931462659584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42597209 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚614238531367936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176613947 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144512507 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42596918 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 321126454 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593389530658304 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2752931821123072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31138617 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚593389512046080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144512505 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20733235239936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2752931462659584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31138620 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845230337866123e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176603804 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20701956509883 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚1469 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2732198568843776© -(è7‹P N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚593389512046080 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144512505 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20733235239936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2752931462659584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31138620 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 % _timestamp‚1.7845230337866123e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176603804 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20701956509883 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚291 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _step‚1469 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2732198568843776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2732198568843776 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2716208224832272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 _runtime‚16207.362187241 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176603804 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2752931462659584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42597209 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚614238531367936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176613947 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144512507 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42596918 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 321126454 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593389530658304 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2752931821123072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31138617 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452½ €µ`«4é7/ _wandbruntime‚16207  _runtime‚16207šn'ê7N RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 321143917 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73739538 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 321133758 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2732346843345920 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321143917 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 321133758 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20850111791616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚593422710928384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20703064898247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176613460 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚17849 % _timestamp‚1.7845230344048004e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144520300 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73739840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144520298 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2753081205731328 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2732346860385280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144520300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20734003914752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20702724336861 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20734345346048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20850004432384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2737058737437351 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20734003914752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144520298 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2716355672539104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176623617 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0š†2Õ ë7•N JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73739538 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 321133758 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2732346843345920 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321143917 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 321133758 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20850111791616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚593422710928384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20703064898247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176613460 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚17849 % _timestamp‚1.7845230344048004e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144520300 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73739840 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.sm7B8yall_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144520298 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2753081205731328 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2732346860385280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144520300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20734003914752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20702724336861 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20734345346048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20850004432384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2737058737437351 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20734003914752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144520298 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2716355672539104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176623617 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 321143917 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20788462917632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2760401115332096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321990611 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2739612293950976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322000756 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42711386 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2760401115332096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2760400756868608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144902155 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20757439723291 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20757099162017 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177098599 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177088456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42711665 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20788462917632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2760401115332096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321990611 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2739612293950976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322000756 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42711386 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2760401115332096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2760400756868608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144902155 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0Á bEå04û7/ _wandbruntime‚16250  _runtime‚16250ûëJÝyü7N FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73940659 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚615987716013056 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2739760585492480 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2739760568453120 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2760550141469696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73940939 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7845230771289167e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2744485754024359 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20758207550269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322008060 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322018219 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177108269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20905991580672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20758548111655 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177108269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2739760568453120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _step‚1474 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144909948 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144909948 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73940659 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚615987716013056 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2739760585492480 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2739760568453120 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2760550141469696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73940939 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 % _timestamp‚1.7845230771289167e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2744485754024359 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20758207550269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322008060 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322018219 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177108269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20905991580672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20758548111655 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177108269 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2739760568453120 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?tÂd4þ7/ _wandbruntime‚16250  _runtime‚16250þK€ÿ7‹ 3loss/SRqqt2/assignment_loss‚0.42812385405801545 3loss/FBt1/assignment_loss‚7.784824239626624e-05 2loss/FBt2/assignment_loss‚0.000892679039507824 1loss/SRqqt1/detection_loss‚0.5915679125207064 trainer/global_step‚17899 epoch‚44 0loss/FRt1/assignment_loss‚1.4964075699658461 /loss/FRt1/detection_loss‚0.4443356914640911 1loss/SRqqt2/detection_loss‚0.6390594970850673 %loss/total_loss‚6.738133812765623 % _timestamp‚1.7845230771306663e+09 0loss/FBt2/detection_loss‚0.29947926895951643 _step‚1475 0loss/FRt2/assignment_loss‚0.5952216704070414 3loss/SRqqt1/assignment_loss‚0.46036278271021686 /loss/FRt2/detection_loss‚1.4697499620133785 /loss/FBt1/detection_loss‚0.3128550762998398 _runtime‚16250.705944064‡\€8 %loss/total_loss‚6.738133812765623 % _timestamp‚1.7845230771306663e+09 0loss/FBt2/detection_loss‚0.29947926895951643 _step‚1475 0loss/FRt2/assignment_loss‚0.5952216704070414 3loss/SRqqt1/assignment_loss‚0.46036278271021686 /loss/FRt2/detection_loss‚1.4697499620133785 /loss/FBt1/detection_loss‚0.3128550762998398 _runtime‚16250.705944064 3loss/SRqqt2/assignment_loss‚0.42812385405801545 3loss/FBt1/assignment_loss‚7.784824239626624e-05 2loss/FBt2/assignment_loss‚0.000892679039507824 1loss/SRqqt1/detection_loss‚0.5915679125207064 trainer/global_step‚17899 epoch‚44 0loss/FRt1/assignment_loss‚1.4964075699658461 /loss/FRt1/detection_loss‚0.4443356914640911 1loss/SRqqt2/detection_loss‚0.6390594970850673à «ÌÍ8‚(:ô ÎÒöÒÞ­¢Ë# proc.memory.rssMB‚ 3025.80078125 network.sent‚ 2.876845e+07% disk./.usageGB‚1168.4348030090332 disk.nvme0n1.in‚ 309.37109375& disk.nvme1n1.out‚48830.29052734375" disk.dm-1.out‚16129.1474609375$ memory_percent‚97.06830978393555& proc.memory.availableMB‚ 480.328125* proc.memory.percent‚18.468022346496582& disk.nvme0n1.out‚103445.1005859375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375 proc.cpu.threads‚17 network.recv‚ 4.234899e+06) disk./.usagePercent‚76.10713584165661 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0( disk.nvme0c0n1.out‚103445.1005859375 cpu‚25.7019809519833" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚48830.29052734375 disk.md0.in‚0 disk.md0.out‚ 718.90625 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚134641.77001953125 disk.rbd0.in‚0ƒÊÿ³ž‚8‚(:“ ÎÒöÒǺ“Ì gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚289.799' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.59966666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=]û·†ƒ8‚(:û ÝÒöÒñʸË( disk.nvme0c0n1.out‚103447.0166015625( disk.nvme1c1n1.out‚48831.04931640625( proc.memory.availableMB‚ 480.33203125 proc.cpu.threads‚17 network.sent‚ 2.8831752e+07 network.recv‚ 4.239655e+06" disk.dm-1.out‚16129.2177734375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚103447.0166015625# proc.memory.rssMB‚ 3025.80078125 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚48831.04931640625 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚134643.45068359375) disk./.usagePercent‚76.10714876230523$ disk./.usageGB‚1168.435001373291 disk.rbd0.out‚0 disk.md0.out‚ 719.33984375$ memory_percent‚97.06828594207764* proc.memory.percent‚18.468022346496582 cpu‚25.701779060795722`H~ž„8‚(:“ ÝÒöÒùÎÌ gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚270.697' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.23233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=V¸…8‚(:ô ìÒöÒš‚ÎË network.recv‚ 4.243261e+06) disk./.usagePercent‚76.10712366643004 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme1c1n1.out‚48831.29541015625$ memory_percent‚97.06828594207764 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚ 719.453125" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚48831.29541015625% disk./.usageGB‚1168.4346160888672" disk.dm-1.out‚16129.2412109375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375( proc.memory.availableMB‚ 480.33203125* proc.memory.percent‚18.468022346496582 network.sent‚ 2.8834716e+07$ disk.dm-0.out‚134644.30908203125& disk.nvme0c0n1.out‚103447.91015625$ disk.nvme0n1.out‚103447.91015625# proc.memory.rssMB‚ 3025.80078125 cpu‚25.7015775673041SGw–ž†8‚(:“ ìÒöÒóµÌ gpu.0.gpu‚56.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚123.236' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.07866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž³é[4‡8/ _wandbruntime‚16293  _runtime‚16293eý§Êˆ8Ä _runtime‚16293.439409948 _step‚1476 & lr-AdamW/pg1‚1.645350800510562e-05 & lr-AdamW/pg2‚1.645350800510562e-05 trainer/global_step‚17949 % _timestamp‚1.7845231198637767e+09w23ω8É trainer/global_step‚17949 % _timestamp‚1.7845231198637767e+09 _runtime‚16293.439409948 _step‚1476 & lr-AdamW/pg1‚1.645350800510562e-05 & lr-AdamW/pg2‚1.645350800510562e-05Ä Œè4Š8/ _wandbruntime‚16293  _runtime‚16293¿š} ‹8†P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2747026036097536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177583251 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2747026019058176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2767870051077632 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177583251 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2767870409541120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74138077 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 322864913 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2747026019058176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚617668335463936 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2730951291579472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177573108 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20812922936699 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42826060 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42826351 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2747026036097536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74138371 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2767870051077632 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177583251 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2767870409541120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74138077 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 322864913 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2747026019058176 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚617668335463936 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2730951291579472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177573108 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20812922936699 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42826060 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42826351 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2747026036097536 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74138371 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74142054 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚596740737936384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2751912770611367 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚16294.059129107 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚596740719324160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322892521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20845483549696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20961791546880 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2731098739286304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31313751 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20845142118400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31313754 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42828303 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145299600 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177592921 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145299598 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20845483549696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177592921 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20845142118400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2747174310599680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322882362 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2768019794149376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 322882362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374YF‚„'8•N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177592921 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145299598 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20845483549696 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177592921 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20845142118400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2747174310599680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 103164928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322882362 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2768019794149376 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 322882362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177582764 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20814031325063 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2768019435678720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 121777152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚292 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177582764 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2768019794149376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 322892521 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145299600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20813690763677 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145299598 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2731098722246944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74142054 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚596740737936384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2751912770611367 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚16294.059129107 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚596740719324160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322892521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20845483549696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20961791546880 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2731098739286304 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31313751 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20845142118400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31313754 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42828303 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145299600Æ ¤qµ¢48/ _wandbruntime‚16294  _runtime‚16294"ú^49‘8Š % _timestamp‚1.7845231204843721e+09 3loss/SRqçp˜×qt2/assignment_loss‚0.43704851264509137 trainer/global_step‚17949 0loss/FRt2/assignment_loss‚0.7048833815837806 3loss/FBt2/assignment_loss‚0.0033633892159863977 epoch‚44 _step‚1479 0loss/FRt1/assignment_loss‚1.5769116380850778 3loss/SRqqt1/assignment_loss‚0.37797905733510007 2loss/FBt1/assignment_loss‚0.017542104788784485 .loss/FRt2/detection_loss‚1.462056325314751 1loss/SRqqt1/detection_loss‚0.5681412570019267 1loss/SRqqt2/detection_loss‚0.6080899671075929 %loss/total_loss‚6.866717386657146 _runtime‚16294.059634828 /loss/FRt1/detection_loss‚0.4645607140051851 /loss/FBt1/detection_loss‚0.3003899061507704 0loss/FBt2/detection_loss‚0.34575113342309904›÷éÇ’8 .loss/FRt2/detection_loss‚1.462056325314751 1loss/SRqqt1/detection_loss‚0.5681412570019267 1loss/SRqqt2/detection_loss‚0.6080899671075929 %loss/total_loss‚6.866717386657146 _runtime‚16294.059634828 /loss/FRt1/detection_loss‚0.4645607140051851 /loss/FBt1/detection_loss‚0.3003899061507704 0loss/FBt2/detection_loss‚0.34575113342309904 % _timestamp‚1.7845231204843721e+09 3loss/SRqqt2/assignment_loss‚0.43704851264509137 trainer/global_step‚17949 0loss/FRt2/assignment_loss‚0.7048833815837806 3loss/FBt2/assignment_loss‚0.0033633892159863977 epoch‚44 _step‚1479 0loss/FRt1/assignment_loss‚1.5769116380850778 3loss/SRqqt1/assignment_loss‚0.37797905733510007 2loss/FBt1/assignment_loss‚0.017542104788784485Ç ø^P“8‚(:ô ûÒöÒÅÞëË disk.dm-0.in‚ 539.2421875* proc.memory.percent‚18.468022346496582 network.recv‚ 4.246765e+06% disk./.usageGB‚1168.4346351623535 disk.rbd0.out‚0) disk.nvme0c0n1.out‚103450.71337890625 disk.nvme1n1.in‚ 229.87109375 proc.cpu.threads‚17 disk.dm-1.out‚ 16129.40625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0" disk.dm-0.out‚134647.185546875# memory_percent‚97.0686674118042( proc.memory.availableMB‚ 480.26953125) disk./.usagePercent‚76.10712490880009 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚103450.71337890625( disk.nvme1c1n1.out‚48832.22314453125# proc.memory.rssMB‚ 3025.80078125 cpu‚25.701376206765055 network.sent‚ 2.8853556e+07 disk.dm-1.in‚0& disk.nvme1n1.out‚48832.22314453125 disk.md0.out‚ 719.765625²V—]œ”8‚(:‘ ûÒöÒÔ·¶Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚267.778' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.25933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#Žƒ~•8‚(:ó ŠÓöÒ‰²èË network.sent‚ 2.8904116e+07) disk./.usagePercent‚75.93016619015314 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375* proc.memory.percent‚18.468022346496582% disk./.usageGB‚1165.7178764343262 disk.md0.in‚0 disk.md0.out‚ 719.97265625" disk.dm-0.out‚134648.201171875 disk.dm-1.out‚ 16129.4453125 disk.rbd0.out‚0) disk.nvme0c0n1.out‚103451.77392578125 disk.nvme0n1.in‚ 309.37109375 network.recv‚ 4.252172e+06 disk.dm-0.in‚ 539.2421875' disk.nvme0n1.out‚103451.77392578125( disk.nvme1c1n1.out‚48832.68212890625 disk.nvme1n1.in‚ 229.87109375 cpu‚25.701160787174246 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme1n1.out‚48832.68212890625$ memory_percent‚97.06878662109375" proc.memory.availableMB‚480.25# proc.memory.rssMB‚ 3025.80078125&áƒHž–8‚(:“ ŠÓöÒÍÙ¬Ì gpu.0.gpu‚92.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚274.742' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.58066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½QŽ¬†—8‚(:û ™ÓöÒÈŽ£Ë& disk.nvme1n1.out‚48833.28271484375 cpu‚25.700993320699588 network.recv‚ 4.254771e+06% disk./.usageGB‚1165.7178421020508 disk.md0.in‚0 disk.md0.out‚ 720.3359375! disk.dm-1.out‚16129.517578125" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0n1.out‚103455.66845703125( disk.nvme1c1n1.out‚48833.28271484375) disk./.usagePercent‚75.93016395388705# disk.dm-0.out‚134651.8330078125 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚97.06881046295166( proc.memory.availableMB‚ 480.24609375# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 disk.dm-0.in‚ 539.2421875 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 proc.cpu.threads‚17 network.sent‚ 2.8906163e+07 disk.rbd0.in‚0) disk.nvme0c0n1.out‚103455.66845703125!Ý㜘8‚(:‘ ™ÓöÒª»öË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.638' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.21266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RæNr4™8/ _wandbruntime‚16336  _runtime‚16336Oçï¡Ìš8Æ _step‚1480 ' lr-AdamW/pg1‚1.5667874107895597e-05 ' lr-AdamW/pg2‚1.5667874107895597e-05 trainer/global_step‚17999 % _timestamp‚1.7845231631652846e+09 _runtime‚16336.742088826†Ž!Ñ›8Ë _runtime‚16336.742088826 _step‚1480 ' lr-AdamW/pg1‚1.5667874107895597e-05 ' lr-AdamW/pg2‚1.5667874107895597e-05 trainer/global_step‚17999 % _timestamp‚1.7845231631652846e+09È ×i&4œ8/ _wandbruntime‚16336  _runtime‚16336ÜüB (8…P BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74339648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚619383228015616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178057760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145681455 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2775339345286656 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2754439761204736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31398717 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2759191248142539 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚17999 $ _timestamp‚1.784523163165999e+09 _runtime‚16336.742196206 _step‚1481 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178067903 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20899601121280 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20899942545408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20899601121280 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2775339703750144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2738322824953072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42940646 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚598366552558080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145681457 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74339363 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178067903 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2754439744165376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31398720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚619383102036992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20899942545408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2775339703750144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145681455 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21016549478912Úɺä(ž8ŠP F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 323739215 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2754439761204736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚598366571170304 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20868065588833 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145681457 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2759190890541905 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20868406150107 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2754439744165376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74339648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚619383228015616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178057760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145681455 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2775339345286656 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2754439761204736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31398717 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2759191248142539 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚17999 $ _timestamp‚1.784523163165999e+09 _runtime‚16336.742196206 _step‚1481 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178067903 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚20899601121280 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20899942545408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20899601121280 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2775339703750144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2738322824953072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42940646 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚598366552558080 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145681457 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74339363 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178067903 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2754439744165376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31398720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚619383102036992 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚20899942545408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2775339703750144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145681455 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21016549478912 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145689248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145689250 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145689248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2754588035706880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42943237 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 323756664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21017666285568 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2759339429597629 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚598399732828160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2775489088358400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2759339787198375 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74343691 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pooÛ ›!l.freed‚ 178067416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74343408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20900711220224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚17999 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2754588035706880 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20900711220224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 323756664 _step‚1482 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178077573 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323766823 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚619417525085184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚619417399113728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚16337.355383409 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145689250 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2775488729887744 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20901052651520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚598399751440384ïNA§'¡8”N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21017773644800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2754588018667520 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31400454 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2738470255620544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 $ _timestamp‚1.784523163779567e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 323766823 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145689248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145689250 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145689248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2754588035706880 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42943237 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 323756664 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21017666285568 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2759339429597629 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚598399732828160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2775489088358400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2759339787198375 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74343691 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178067416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74343408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20900711220224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚17999 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2754588035706880 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20900711220224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 323756664 _step‚1482 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178077573 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 323766823 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚619417525085184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚619417399113728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚16337.355383409 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145689250 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2775488729887744 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20901052651520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚598399751440384 Q&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6619706153869629 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4519205689430237 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.231391042470932 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 'particle/accuracy‚0.853380024433136 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚17999 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9903878569602966 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2758072316646576 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21890856325626373 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 *jet/accuracy_1_of_2‚0.7710131406784058 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09888458251953125 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9108358025550842 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 particle/accuracy_3_of_5‚1 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.594857931137085 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05061373859643936 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_4‚0.9672623872756958 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5952070951461792 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4794606566429138 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2901264727115631 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9971181154251099 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07142780721187592 +jet/accuracy_3_of_3‚0.07508111000061035 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3025747835636139 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9460712671279907 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30090317130088806 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08010600507259369 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6847485303878784 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 *jet/accuracy_1_of_3‚0.9184781908988953 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3386167883872986 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.29088011384010315 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08096124976873398 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29397180676460266 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8641345500946045 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6412179470062256 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07142780721187592 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6219189763069153 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34632906317710876 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4577919542789459 *jet/accuracy_2_of_4‚0.7346791625022888 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20552773773670197 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8690111041069031 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 _step‚1484 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.039652176201343536 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05119020491838455 jet/accuracy_1_of_5‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6556620597839355 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 _runtime‚16353.539489862 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26521429419517517 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7724490165710449 'particle/f_score‚0.6532965898513794 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5771148204803467 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7087634205818176 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 epoch‚44 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5951648354530334 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8389833569526672 +particle/sensitivity‚0.6432920098304749 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21938930451869965 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9579095244407654 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6350611448287964 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5754294395446777 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22868740558624268 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2559680640697479 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2818753719329834 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5792859196662903 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_2_of_3‚0.5590853095054626 )loss/val_total_loss‚7.429218927024791 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.909355640411377 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.29430854320526123 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8840345740318298 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.621479332447052 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6219990849494934 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3188396394252777 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20944622159004211 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24036605656147003 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07834268361330032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6667923331260681 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2642160952091217 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3474518060684204 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6462135910987854 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7546446323394775 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5770832896232605 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8410053253173828 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6840260624885559 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34283050894737244 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4813595712184906 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7283516526222229 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.15982067584991455 jet/accuracy_5_of_5‚0 % _timestamp‚1.7845231799634607e+09 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3010045886039734 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_1‚0.3314977288246155 particle/accuracy_3_of_4‚1 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6636435389518738 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4716132581233978 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5815136432647705 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5738338232040405 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.061373937875032425 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6487011909484863 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09142044186592102 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3026657700538635 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2901264727115631 7validation_average_jet_accuracy‚0.37216389179229736 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974238276481628 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6190701723098755 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7067965865135193 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27536994218826294 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.575310468673706 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.646770179271698 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1189260333776474 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 particle/accuracy_4_of_5‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.621859073638916 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5643208622932434 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6777795553207397 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2283099889755249 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07887955009937286 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.706414520740509 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7724490165710449 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7095591425895691 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5754294395446777 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29789501428604126 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6150737404823303 particle/accuracy_2_of_2‚1 /particle/accuracy_3_of_3‚0.9994120597839355 particle/accuracy_1_of_5‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2831750214099884 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9907387495040894 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172ÄÇýþƒ©8Òz >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRq5ÅËÕ<qt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6847485303878784 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 *jet/accuracy_1_of_3‚0.9184781908988953 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3386167883872986 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.29088011384010315 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08096124976873398 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29397180676460266 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8641345500946045 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6412179470062256 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07142780721187592 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6219189763069153 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34632906317710876 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4577919542789459 *jet/accuracy_2_of_4‚0.7346791625022888 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20552773773670197 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8690111041069031 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 _step‚1484 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.039652176201343536 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.05119020491838455 jet/accuracy_1_of_5‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6556620597839355 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 _runtime‚16353.539489862 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26521429419517517 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7724490165710449 'particle/f_score‚0.6532965898513794 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5771148204803467 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7087634205818176 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 epoch‚44 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5951648354530334 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8389833569526672 +particle/sensitivity‚0.6432920098304749 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21938930451869965 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9579095244407654 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6350611448287964 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5754294395446777 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22868740558624268 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2559680640697479 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2818753719329834 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5792859196662903 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_2_of_3‚0.5590853095054626 )loss/val_total_loss‚7.429218927024791 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.909355640411377 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.29430854320526123 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8840345740318298 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.621479332447052 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6219990849494934 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3188396394252777 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20944622159004211 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9968472719192505 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24036605656147003 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07834268361330032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6667923331260681 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2642160952091217 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3474518060684204 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6462135910987854 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7546446323394775 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5770832896232605 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8410053253173828 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9255189299583435 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6840260624885559 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34283050894737244 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4813595712184906 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7283516526222229 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.15982067584991455 jet/accuracy_5_of_5‚0 % _timestamp‚1.7845231799634607e+09 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3010045886039734 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_1‚0.3314977288246155 particle/accuracy_3_of_4‚1 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6636435389518738 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4716132581233978 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5815136432647705 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5738338232040405 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.061373937875032425 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6487011909484863 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09142044186592102 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3026657700538635 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2901264727115631 7validation_average_jet_accuracy‚0.37216389179229736 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9974238276481628 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6190701723098755 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7067965865135193 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27536994218826294 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.575310468673706 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.646770179271698 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.1189260333776474 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 particle/accuracy_4_of_5‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.621859073638916 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5643208622932434 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6777795553207397 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2283099889755249 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07887955009937286 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.706414520740509 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7724490165710449 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7095591425895691 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.983164370059967 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5754294395446777 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29789501428604126 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6150737404823303 particle/accuracy_2_of_2‚1 /particle/accuracy_3_of_3‚0.9994120597839355 particle/accuracy_1_of_5‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2831750214099884 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9907387495040894 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.732887327671051 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2718464136123657 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5974946022033691 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5047450661659241 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.997355580329895 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6415666937828064 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7148427367210388 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6619706153869629 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4519205689430237 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.231391042470932 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 'particle/accuracy‚0.853380024433136 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚17999 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9903878569602966 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2758072316646576 particle/accuracy_2_of_3‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925479888916016 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21890856325626373 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 *jet/accuracy_1_of_2‚0.7710131406784058 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9915822148323059 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09888458251953125 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9108358025550842 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 particle/accuracy_3_of_5‚1 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.594857931137085 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05061373859643936 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_4‚0.9672623872756958 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9976599812507629 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5952070951461792 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4794606566429138 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2901264727115631 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9971181154251099 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07142780721187592 +jet/accuracy_3_of_3‚0.07508111000061035 particle/accuracy_1_of_4‚1 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3025747835636139 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9460712671279907 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30090317130088806 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08010600507259369Ì Ô…‡ª8‚(:ü ·ÓöÒ¦½ÂË$ disk.dm-0.out‚134663.76416015625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0n1.out‚103466.72998046875( disk.nvme1c1n1.out‚48835.63720703125$ disk./.usageGB‚1165.717586517334& disk.nvme1n1.out‚48835.63720703125$ memory_percent‚97.06923961639404# proc.memory.rssMB‚ 3025.80078125 disk.dm-1.in‚0# disk.dm-1.out‚16129.72607421875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375( proc.memory.availableMB‚ 480.17578125* proc.memory.percent‚18.468022346496582 cpu‚25.70344778433906 proc.cpu.threads‚17 network.sent‚ 2.8973552e+07 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 720.9140625 disk.dm-0.in‚ 539.2421875) disk.nvme0c0n1.out‚103466.72998046875" disk.nvme1c1n1.in‚ 229.87109375 network.recv‚ 4.263361e+06) disk./.usagePercent‚75.93014730612825 disk.md0.in‚0X±ÇSž«8‚(:“ ·ÓöÒ®Ÿ„Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚160.171' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚53.39033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dvÔ.y¬8‚(:î ÆÓöÒƧˆÌ$ disk.dm-0.out‚134665.39208984375# disk.dm-1.out‚16129.79248046875' disk.nvme0c0n1.out‚103468.615234375 proc.cpu.threads‚17$ disk./.usageGB‚1165.718376159668 disk.nvme1n1.out‚ 48836.09375 disk.md0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375* proc.memory.percent‚18.468022346496582 network.sent‚ 2.9032364e+07 disk.nvme1n1.in‚ 229.87109375 disk.rbd0.in‚0% disk.nvme0n1.out‚103468.615234375" disk.nvme1c1n1.out‚ 48836.09375$ memory_percent‚97.07071781158447( proc.memory.availableMB‚ 479.93359375 cpu‚25.703214128093897 network.recv‚ 4.268462e+06) disk./.usagePercent‚75.93019874024868 disk.md0.out‚ 721.21484375 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0" disk.nvme1c1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3025.80078125çóÀìž­8‚(:“ ÆÓöÒלּË gpu.0.gpu‚65.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.202' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.06733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"0îÜ„®8‚(:ù ÕÓöÒãÌ¡Ì disk.md0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375# proc.memory.rssMB‚ 3025.80078125 proc.cpu.threads‚17% disk./.usageGB‚1165.7184791564941& disk.nvme1n1.out‚48837.00341796875 disk.md0.out‚ 721.66015625 disk.rbd0.in‚0( disk.nvme1c1n1.out‚48837.00341796875* proc.memory.percent‚18.468022346496582 network.recv‚ 4.271828e+06 disk.nvme1n1.in‚ 229.87109375$ disk.dm-0.out‚134668.23974609375" disk.dm-1.out‚16129.8974609375 disk.rbd0.out‚0$ memory_percent‚97.06921577453613' proc.memory.availableMB‚ 480.1796875 cpu‚25.702982038954204 network.sent‚ 2.9035328e+07) disk./.usagePercent‚75.93020544904701 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0' disk.nvme0c0n1.out‚103471.626953125% disk.nvme0n1.out‚103471.626953125" disk.nvme1c1n1.in‚ 229.87109375=kF“¯8‚(:ˆ ÕÓöÒâÓË gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚122.106' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹A1û4°8/ _wandbruntime‚16395  _runtime‚16395)›ãŒÌ±8Æ ' lr-AdamW/pg1‚1.4900967646059352e-05 ' lr-AdamW/pg2‚1.4900967646059352e-05 trainer/global_step‚18049 % _timestamp‚1.7845232224047585e+09 _runtime‚16395.982037585 _step‚1485X >¸Ñ²8Ë ' lr-AdamW/pg1‚1.4900967646059352e-05 ' lr-AdamW/pg2‚1.4900967646059352e-05 trainer/global_step‚18049 % _timestamp‚1.7845232224047585e+09 _runtime‚16395.982037585 _step‚1485Í MS~4³8/ _wandbruntime‚16395  _runtime‚16395©8üi (´8…P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31489533 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146193915 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 _runtime‚16395.982099916 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178623906 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43090328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚600048163336704 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20975829483008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚621141395764224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚621141282368512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2784771301207040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2784771659670528 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146193915 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178623906 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146193913 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2763795830187520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2784771301207040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20975488058880 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74579864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚600048144724480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178623906 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146193913 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2763795830187520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2784771301207040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚20975488058880 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74579864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚600048144724480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113395712 _runtime‚16395.982099916 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178623906 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43090328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚600048163336704 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚20975829483008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚621141395764224 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚621141282368512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2784771301207040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2784771659670528 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146193915 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324845427 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146201708 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2763944087650304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178643719 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20976598157824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21094354906624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20976939589120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2784920685808128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21094264324608 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74583564 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚600081343606784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 _runtime‚16396.595255391 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7845232230194612e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2763944087650304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20976598157824 _step‚1487 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚600081324994560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324845427 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2763944104689664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20976939589120 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2747763062041728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2763944104689664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2784921044278784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2784921044278784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74583847 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2747763045002368 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚18049 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178633562 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31491267 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20945301600776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20944961039390 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚621175698513408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43092577 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146201706Ýêu'¸8”N VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43092577 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146201706 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324845427 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146201708 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2763944087650304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178643719 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚20976598157824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21094354906624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚20976939589120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2784920685808128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21094264324608 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74583564 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚600081343606784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 _runtime‚16396.595255391 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7845232230194612e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2763944087650304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚20976598157824 _step‚1487 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚600081324994560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324845427 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2763944104689664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚20976939589120 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2747763062041728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2763944104689664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2784921044278784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90582016 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2784921044278784 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74583847 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2747763045002368 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 trainer/global_step‚18049 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178633562 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31491267 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚20945301600776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚20944961039390 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚621175698513408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ï ®âZ44¹8/ _wandbruntime‚16396  _runtime‚16396O ‹4º8Š 0loss/FRt1/detection_loss‚0.49210079340476415 1loss/SRqqt2/detection_loss‚0.6628569134843089 0loss/FRt1/assignment_loss‚1.4755675785141846 3loss/SRqqt2/assignment_loss‚0.40480569753310003 /loss/FRt2/detection_loss‚1.5711123303205556 0loss/FBt1/detection_loss‚0.27568951040969597 _runtime‚16396.596692944 _step‚1488 /loss/FRt2/assignment_loss‚0.747179639192275 3loss/FBt1/assignment_loss‚5.664244908732346e-05 1loss/FBt2/assignment_loss‚0.01404741039004848 %loss/total_loss‚6.994889677782453 epoch‚45 % _timestamp‚1.7845232230212877e+09 2loss/SRqqt1/assignment_loss‚0.4312267361715641 1loss/SRqqt1/detection_loss‚0.5939548503730473 0loss/FBt2/detection_loss‚0.32629157553982135 trainer/global_step‚18049ˆ â¡»8 /loss/FRt2/detection_loss‚1.5711123303205556 0loss/FBt1/detection_loss‚0.27568951040969597 _runtime‚16396.596692944 _step‚1488 /loss/FRt2/assignment_loss‚0.747179639192275 3loss/FBt1/assignment_loss‚5.664244908732346e-05 1loss/FBt2/assignment_loss‚0.01404741039004848 %loss/total_loss‚6.994889677782453 epoch‚45 % _timestamp‚1.7845232230212877e+09 2loss/SRqqt1/assignment_loss‚0.4312267361715641 1loss/SRqqt1/detection_loss‚0.5939548503730473 0loss/FBt2/detection_loss‚0.32629157553982135 trainer/global_step‚18049 0loss/FRt1/detection_loss‚0.49210079340476415 1loss/SRqqt2/detection_loss‚0.6628569134843089 0loss/FRt1/assignment_loss‚1.4755675785141846 3loss/SRqqt2/assignment_loss‚0.40480569753310003Ð *Ñ»…¼8‚(:ú äÓöÒ›“Ì network.sent‚ 2.9054169e+07" disk.dm-1.out‚16129.9794921875' disk.nvme0n1.out‚103473.27392578125& disk.nvme1n1.out‚48837.68603515625 disk.md0.out‚ 722.078125# proc.memory.rssMB‚ 3025.80078125 disk.nvme0n1.in‚ 309.37109375) disk./.usagePercent‚75.99562194754647$ disk./.usageGB‚1166.722785949707 disk.dm-1.in‚0 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0$ memory_percent‚97.06943035125732( disk.nvme1c1n1.out‚48837.68603515625( proc.memory.availableMB‚ 480.14453125* proc.memory.percent‚18.468022346496582 cpu‚25.702720949344094 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 309.37109375) disk.nvme0c0n1.out‚103473.27392578125 network.recv‚ 4.275282e+06 disk.dm-0.in‚ 539.2421875# disk.dm-0.out‚134669.6142578125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 229.871093750Dn5ž½8‚(:“ äÓöÒà®ßÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚264.739' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.24633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿŸY-ž¾8‚(:“ óÓöÒ´þÖÌ gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚283.927' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.64233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TzO‚¿8‚(:÷ óÓöÒÓ¸ªÍ) disk./.usagePercent‚76.11008647054632 disk.md0.out‚ 722.59765625 disk.dm-0.in‚ 539.2421875( proc.memory.availableMB‚ 480.12109375# proc.memory.rssMB‚ 3025.80078125 proc.cpu.threads‚17 network.recv‚ 4.280248e+06 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚103477.68115234375' disk.nvme1c1n1.out‚50752.0380859375 disk.md0.in‚0$ disk.dm-0.out‚136587.18212890625 disk.dm-1.out‚ 16130.078125 disk.rbd0.out‚0 network.sent‚ 2.9105028e+07" disk.nvme1c1n1.in‚ 229.87109375" disk.nvme0c0n1.in‚ 309.37109375) disk.nvme0c0n1.out‚103477.68115234375$ memory_percent‚97.06957340240479% disk./.usageGB‚1168.4801025390625 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚50752.0380859375 disk.dm-1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚18.468022346496582 cpu‚25.70247129622847—?—Z4À8/ _wandbruntime‚16438  _runtime‚16438Ž1vÊÁ8Ä _runtime‚ 16438.7807655 _step‚1489 ' lr-AdamW/pg1‚1.4152837876566029e-05 ' lr-AdamW/pg2‚1.4152837876566029e-05 trainer/global_step‚18099 % _timestamp‚1.7845232652050927e+09—¦ÓÏÂ8É ' lr-AdamW/pg1‚1.4152837876566029e-05 ' lr-AdamW/pg2‚1.4152837876566029e-05 trainer/global_step‚18099 % _timestamp‚1.7845232652050927e+09 _runtime‚ 16438.7807655 _step‚1489Ñ /Tµ9(Ã8/ _wandbruntime‚16438  _runË[  time‚16438òñ5‘ (Ä8…P ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚622856368270336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7845232652061284e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146583563 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2754987147708496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚601707158228480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20999335864546 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43205386 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚622856256971776 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2771209538255360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179108558 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2792240595416064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚1490 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325692121 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21031057160704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146583563 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2771209538255360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚18099 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74781619 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146583565 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21149191429632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179118701 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179108558 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 325702266 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2771209555294720 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2771209555294720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2792240595416064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20999676425820 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21031057160704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2775986483573042 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31576233 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31576236 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43205670 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179118701 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74781906 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21031398584832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31576236 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43205670 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179118701 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74781906 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21031398584832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146583563 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2754987147708496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚601707158228480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚20999335864546 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43205386 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚622856256971776 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2771209538255360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179108558 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2792240595416064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚1490 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 325692121 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21031057160704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146583563 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2771209538255360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚18099 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74781619 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146583565 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21149191429632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179118701 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179108558 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 325702266 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2771209555294720 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2771209555294720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2792240595416064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚20999676425820 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21031057160704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2775986483573042 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31576233 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325719729 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43207744 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179128371 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31577967 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21032508690944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179118214 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚622890669696512 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179128371 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚18099 _runtime‚16439.394464248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2776135022628766 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2755134578375968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74785986 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚601740357110784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146591356 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146591358 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325709570 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2792389980017152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2792390338487808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21150209420800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2755134595415328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21032167259648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31577970 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43208016 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚622890547919360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21000444252798 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325709570 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2776135380229512 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74785711 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21000784814184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2771357812757504 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179118214 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21150312585728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 % _timestamp‚1.7845232658186896e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2771357829796864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374Ïjü¹eÈ8•N E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21032167259648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31577970 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43208016 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚622890547919360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21000444252798 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 325709570 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2776135380229512 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74785711 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21000784814184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 L>DeviceStatsMonitor.on_train_«Ý­o¶batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2771357812757504 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179118214 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21150312585728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 % _timestamp‚1.7845232658186896e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2771357829796864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 325719729 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43207744 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179128371 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31577967 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21032508690944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179118214 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚622890669696512 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179128371 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚18099 _runtime‚16439.394464248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2776135022628766 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2755134578375968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74785986 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚601740357110784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146591356 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146591358 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 325709570 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2792389980017152 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2792390338487808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21150209420800 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2755134595415328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528Ó Ñü6Ÿ4É8/ _wandbruntime‚16439  _runtime‚16439L²%ä Ê8‡ _runtime‚16439.394998078 /loss/FBt2/detection_loss‚0.1997754251690192 %loss/total_loss‚6.535139694165435 0loss/FRt2/assignment_loss‚0.4937791148638396 2loss/SRqqt2/assignment_loss‚0.3906089965327446 2loss/FBt2/assignment_loss‚0.004878747226218804 trainer/global_step‚18099 2loss/FBt1/assignment_loss‚0.009325910171911313 .loss/FRt2/detection_loss‚1.370635935135426 1loss/SRqqt2/detection_loss‚0.5743226442878838 % _timestamp‚1.7845232658197129e+09 _step‚1492 /loss/FRt1/assignment_loss‚1.563298533408528 3loss/SRqqt1/assignment_loss‚0.39912002701380805 /loss/FRt1/detection_loss‚0.4827537192266279 1loss/SRqqt1/detection_loss‚0.6978627803459033 0loss/FBt1/detection_loss‚0.34877786078352546 epoch‚45¤`ßêË8Œ 3loss/SRqqt1/assignment_loss‚0.39912002701380805 /loss/FRt1/detection_loss‚0.4827537192266279 1loss/SRqqt1/detection_loss‚0.6978627803459033 0loss/FBt1/detection_loss‚0.34877786078352546 epoch‚45 _runtime‚16439.394998078 /loss/FBt2/detection_loss‚0.1997754251690192 %loss/total_loss‚6.535139694165435 0loss/FRt2/assignment_loss‚0.4937791148638396 2loss/SRqqt2/assignment_loss‚0.3906089965327446 2loss/FBt2/assignment_loss‚0.004878747226218804 trainer/global_step‚18099 2loss/FBt1/assignment_loss‚0.009325910171911313 .loss/FRt2/detection_loss‚1.370635935135426 1loss/SRqqt2/detection_loss‚0.5743226442878838 % _timestamp‚1.7845232658197129e+09 _step‚1492 /loss/FRt1/assignment_loss‚1.563298533408528Ô ÅaÍÂ~Ì8‚(:ó ‚ÔöÒÛ­íË cpu‚25.702215624858912 proc.cpu.threads‚17( disk.nvme1c1n1.out‚50752.65673828125& disk.nvme1n1.out‚50752.65673828125 network.sent‚ 2.912083e+07) disk./.usagePercent‚76.11009616103279 disk.dm-1.out‚ 16130.140625( proc.memory.availableMB‚ 479.92578125* proc.memory.percent‚18.468022346496582 network.recv‚ 4.283806e+06 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.nvme1n1.in‚ 229.87109375% disk./.usageGB‚1168.4802513122559 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚103478.6005859375 disk.nvme0n1.in‚ 309.37109375& disk.nvme0n1.out‚103478.6005859375 disk.md0.in‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 723.03125 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚136587.72509765625# memory_percent‚97.0707654953003# proc.memory.rssMB‚ 3025.80078125mæ(1‘Í8‚(:† ‚ÔöÒ‚ƳÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚120.591' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.197 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,S²Þ“Î8‚(:ˆ ‘ÔöÒÏý²Ë gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚282.696' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C}Š Ï8‚(:ö ‘ÔöÒÿ¤‹Ì disk.nvme0n1.in‚ 309.37109375 disk.md0.in‚0) disk./.usagePercent‚76.10707347467967% disk./.usageGB‚1168.4338455200195& disk.nvme1c1n1.out‚50992.361328125$ disk.dm-0.out‚137458.57373046875 disk.dm-1.in‚0" disk.dm-1.out‚16130.3251953125( proc.memory.availableMB‚ 479.80859375# proc.memory.rssMB‚ 3025.80078125" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875# memory_percent‚97.0714807510376 cpu‚25.70193963908254 proc.cpu.threads‚17' disk.nvme0n1.out‚104110.64111328125 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 723.359375* proc.memory.percent‚18.468022346496582$ disk.nvme1n1.out‚50992.361328125 network.sent‚ 2.9172394e+07 network.recv‚ 4.288834e+06 disk.rbd0.in‚0 disk.rbd0.out‚0) disk.nvme0c0n1.out‚104110.64111328125“I ÂÐ8‚(:÷  ÔöÒôîÜË disk.dm-1.in‚0' disk.nvme0c0n1.out‚104111.619140625 disk.nvme0n1.in‚ 309.37109375% disk.nvme0n1.out‚104111.619140625" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 723.5234375 disk.rbd0.in‚0& disk.nvme1n1.out‚50992.69482421875* proc.memory.percent‚18.468022346496582 cpu‚25.701666101858667 proc.cpu.threads‚17 network.sent‚ 2.9177266e+07$ disk.dm-0.out‚137459.46826171875" disk.dm-1.out‚16130.3642578125" disk.nvme0c0n1.in‚ 309.37109375$ memory_percent‚97.07150459289551' proc.memory.availableMB‚ 479.8046875 network.recv‚ 4.292617e+06% disk./.usageGB‚1168.4340515136719 disk.dm-0.in‚ 539.2421875 disk.rbd0.out‚0( disk.nvme1c1n1.out‚50992.69482421875# proc.memory.rssMB‚ 3025.80078125( disk./.usagePercent‚76.1070868922763 disk.md0.in‚0+KööÑ8‚(:’  ÔöÒǾ¯Ì gpu.0.gpu‚12.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.922' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.30733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•lCB4Ò8/ _wandbruntime‚16482  _runtime‚16482A¢ÄÌÓ8Æ ' lr-AdamW/pg2‚1.3423532850392653e-05 trainer/global_step‚18149 % _timestamp‚1.7845233085321176e+09 _runtime‚16482.107720313 _step‚1493 ' lr-AdamW/pg1‚1.3423532850392653e-05¼`ÑÔ8Ë trainer/global_step‚18149 % _timestamp‚1.7845233085321176e+09 _runtime‚16482.107720313 _step‚1493 ' lr-AdamW/pg1‚1.3423532850392653e-05 ' lr-AdamW/pg2‚1.3423532850392653e-05Õ ¯¼hY4Õ8/ _wandbruntime‚16482  _runtime‚16482Øì¿Ð (Ö8†P XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21086626262528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2778623280401920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326576568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21204950936576 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2762358698121456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚16482.108517315 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2778623280401920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31662933 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2799709889625088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚624571122669056 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146973213 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179593210 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚624571246550528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚18149 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2778623263362560׈)4!×8‹P ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚624571122669056 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146973213 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179593210 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚624571246550528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚18149 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2778623263362560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21086626262528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2778623280401920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326576568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21204950936576 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2762358698121456 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _runtime‚16482.108517315 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2778623280401920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31662933 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2799709889625088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344Ö / ê4Ø8/ _wandbruntime‚16482  _runtime‚16482å 'Ù8N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326583872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146981006 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2799859632696832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21088077792768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚624605536419840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚624605418836992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1495 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21055927466206 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74987547 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326594031 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31664670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21088077792768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2783562039215774 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2762506111749568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _runtime‚16482.724935343 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2778771554904064 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2783562396816520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146981008 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179602866 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146981006 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43322880 trainer/global_step‚18149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326594031 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2799859274226176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845233091490762e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2778771537864704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179602866 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ôñ·¼'Ú8”N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21087736361472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31664667 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146981008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2799859632696832 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179613023 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2778771554904064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21087736361472 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326583872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74987842 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21206165805056 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179602866 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326583872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146981006 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2799859632696832 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21088077792768 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚624605536419840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚624605418836992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1495 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21055927466206 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74987547 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326594031 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31664670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21088077792768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚295 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2783562039215774 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2762506111749568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 _runtime‚16482.724935343 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2778771554904064 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2783562396816520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146981008 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179602866 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146981006 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43322880 trainer/global_step‚18149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326594031 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2799859274226176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845233091490762e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2778771537864704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 EÉ“è8„P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 327440725 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21142536788480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2807179183834112 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147362865 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2786036988469760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2807179542297600 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2807179183834112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚18199 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180077862 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 327450870 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2769730231495056 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2769730214455696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚297 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2786037005509120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43436490 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚605025203848704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21260902712320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147362863 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21142195364352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚605025185236480 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.¤‹ŸAwall.freed‚2790840516747058 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147362863 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180077862 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21142536788480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21260799540224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31749633 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75186126 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21110642852636 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2807179542297600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2786037005509120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75185826 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 # _timestamp‚1.78452335100011e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180088005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43436193 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21110302291362 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚626286106561024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _step‚1498 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _runtime‚16524.579026817 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21142195364352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2807179542297600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2786037005509120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75185826 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 # _timestamp‚1.78452335100011e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180088005 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43436193 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21110302291362 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚626286106561024 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _step‚1498 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _runtime‚16524.579026817 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21142195364352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2807179183834112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚18199 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488Ú WyÓ›4ê8/ _wandbruntime‚16525  _runtime‚165258#Ãn'ë8N P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180097675 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605058384118784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21262021552128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21143646894592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75190109 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚18199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 327458174 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21143646894592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2790989413403528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21143305463296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2807328568435200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327468333 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2786185280011264 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75189820 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147370656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845233519474003e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 _runtime‚16525.523180233 _step‚1499 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2769877645123168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2786185262971904 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180087518 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180087518 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327468333 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚626320405670912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21111751241000 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147370658 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21111410679614 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43438739áquŸ ì8•N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚626320277602304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147370656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21143305463296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2769877662162528 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 327458174 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31751367 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21261912095744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180097675 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚605058365506560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2790989413403528 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21143305463296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2807328568435200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 109456384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327468333 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2786185280011264 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75189820 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147370656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845233519474003e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 _runtime‚16525.523180233 _step‚1499 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2769877645123168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2786185262971904 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180087518 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180087518 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327468333 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚626320405670912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21111751241000 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147370658 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMon^ë-‘ itor.on_train_batch_end/requested_bytes.small_pool.freed‚21111410679614 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43438739 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180097675 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605058384118784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21262021552128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21143646894592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75190109 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚18199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 327458174 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21143646894592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚308 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31836336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚606684198740480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180572657 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21198105890304 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43551313 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21197764466176 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180562514 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21166126066044 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1502 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43551618 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2777101764868656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75387646 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚628000952384000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21316735031296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2777101747829296 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2793450730616320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2814648836506624 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2814648478043136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31836333 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚606684217352704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚308 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31836336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚606684198740480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180572657 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21198105890304 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43551313 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21197764466176 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180562514 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21166126066044 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1502 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43551618 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2777101764868656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75387646 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚628000952384000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21316735031296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2777101747829296 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2793450730616320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2814648836506624 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2814648478043136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31836333 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚606684217352704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2777249178496768 _step‚1503 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328342635 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21198874565120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21166893893022 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚628035242122240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147760306 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21199215996416 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147760306 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31838070 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2777249195536128 OADeviceStatsMonitor.on_train_batch_end/active)Ù¯–.large_pool.allocated‚ 147760308 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147760308 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚606717397622784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21317844499456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 trainer/global_step‚18249 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 328332476 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21198874565120 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚628035116150784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚16567.875837302 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180572170 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21199215996416 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2814797862644224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2814798221114880 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328342635 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75391649 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2793599005118464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43553582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784523394300069e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2798416429990536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚606717379010560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21198874565120 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚628035116150784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _runtime‚16567.875837302 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180572170 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21199215996416 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2814797862644224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2814798221114880 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328342635 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75391649 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2793599005118464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43553582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784523394300069e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2798416429990536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚606717379010560 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2777249178496768 _step‚1503 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328342635 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21198874565120 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21166893893022 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚628035242122240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147760306 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21199215996416 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147760306 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31838070 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2777249195536128 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147760308 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147760308 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488ß ¤è4ÿ8/ _wandbruntime‚16567  _runtime‚16567<†÷R€9Ž 3loss/SRqqt2/assignment_loss‚0.43475307023811577 %loss/total_loss‚6.890599815975365 % _timestamp‚1.7845233943010485e+09 _runtime‚16567.876377413 0loss/FBt1/detection_loss‚0.43571637069299124 epoch‚45 0loss/FRt1/assignment_loss‚1.4825409723589054 0loss/FRt2/assignment_loss‚0.6924700436832236 2loss/SRqqt1/assignment_loss‚0.3687152802789941 4loss/FBt1/assignment_loss‚1.2038474664812461e-05 /loss/FRt2/detection_loss‚1.5062126015364337 1loss/SRqqt1/detection_loss‚0.6360612587630248 3loss/FBt2/assignment_loss‚0.0032749446102182667 1loss/SRqqt2/detection_loss‚0.5768351161400886 trainer/global_step‚18249 0loss/FBt2/detection_loss‚0.28997309669102833 0loss/FRt1/detection_loss‚0.46403502250767675 _step‚1504<ñ@˜9“ 3loss/FBt2/assignment_loss‚0.0032749446102182667 1loss/SRqqt2/detection_loss‚0.5768351161400886 trainer/global_step‚18249 0loss/FBt2/detection_loss‚0.28997309669102833 0loss/FRt1/detection_loss‚0.46403502250767675 _step‚1504 3loss/SRqqt2/assignment_loss‚0.43475307023811577 %loss/total_loss‚6.890599815975365 % _timestamp‚1.7845233943010485e+09 _runtime‚16567.876377413 0loss/FBt1/detection_loss‚0.43571637069299124 epoch‚45 0loss/FRt1/assignment_loss‚1.4825409723589054 0loss/FRt2/assignment_loss‚0.6924700436832236 2loss/SRqqt1/assignment_loss‚0.3687152802789941 4loss/FBt1/assignment_loss‚1.2038474664812461e-05 /loss/FRt2/detection_loss‚1.5062126015364337 1loss/SRqqt1/detection_loss‚0.6360612587630248à ‚®É‚9‚(:ô ‰ÕöÒìÿ´Ë disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0& disk.nvme0n1.out‚104514.6923828125$ disk.nvme1c1n1.out‚ 51074.8046875# memory_percent‚97.0750093460083* proc.memory.percent‚18.468022346496582 cpu‚25.699923730838748 proc.cpu.threads‚17 network.recv‚ 4.319964e+06) disk./.usagePercent‚76.06380073163471 disk.rbd0.in‚0( disk.nvme0c0n1.out‚104514.6923828125# proc.memory.rssMB‚ 3025.80078125 network.sent‚ 2.9324772e+07" disk.nvme1n1.out‚ 51074.8046875" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375( proc.memory.availableMB‚ 479.23046875% disk./.usageGB‚1167.7695007324219 disk.md0.out‚ 726.55078125 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚137937.39208984375 disk.dm-1.in‚0# disk.dm-1.out‚16131.13720703125üä|—žƒ9‚(:“ ‰ÕöÒ½º¥Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚125.843' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.94766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÊî|„9‚(:ñ ˜ÕöÒîÞáË( disk.nvme0c0n1.out‚104544.9423828125# memory_percent‚97.0750093460083* proc.memory.percent‚18.468022346496582 network.recv‚ 4.325439e+06 disk.dm-1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 3025.80078125 cpu‚25.699726956355775) disk./.usagePercent‚76.06382458513984$ disk.dm-0.out‚137995.20458984375 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375" disk.nvme1n1.out‚ 51102.8984375 proc.cpu.threads‚17 disk.md0.in‚0 disk.md0.out‚ 726.765625 disk.dm-0.in‚ 539.2421875& disk.nvme0n1.out‚104544.9423828125$ disk.nvme1c1n1.out‚ 51102.8984375( proc.memory.availableMB‚ 479.23046875 network.sent‚ 2.938933e+07% disk./.usageGB‚1167.7698669433594# disk.dm-1.out‚16131.19189453125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375_äÔž…9‚(:“ ˜ÕöÒ„¥Ì gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚266.181' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚88.72699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%ºm±|†9‚(:ñ §ÕöÒÙïË$ disk./.usageGB‚1167.769905090332" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 727.11328125" disk.dm-0.out‚138017.751953125 disk.dm-1.in‚0# disk.dm-1.out‚16131.33154296875 disk.rbd0.out‚0' disk.nvme0n1.out‚104567.36572265625 network.recv‚ 4.328652e+06 disk.nvme1n1.in‚ 229.87109375 cpu‚25.6995484236843 proc.cpu.threads‚17 network.sent‚ 2.9392252e+07) disk./.usagePercent‚76.06382706987995$ disk.nvme1c1n1.out‚ 51103.9140625 disk.md0.in‚0" disk.nvme0c0n1.in‚ 309.37109375) disk.nvme0c0n1.out‚104567.36572265625# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582" disk.nvme1n1.out‚ 51103.9140625 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375$ memory_percent‚97.07503318786621' proc.memory.availableMB‚ 479.2265625I¦´Ä“‡9‚(:ˆ §ÕöÒÅŒ´Ë gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚260.109' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚86.703 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[è®z4ˆ9/ _wandbruntime‚16609  _runtime‚16609jo»Ì‰9Æ ' lr-AdamW/pg1‚1.1349028587418166e-05 ' lr-AdamW/pg2‚1.1349028587418166e-05 trainer/global_step‚18299 % _timestamp‚1.7845234361263194e+09 _runtime‚16609.701942825 _step‚1505ñ 0YÑŠ9Ë ' lr-AdamW/pg1‚1.1349028587418166e-05 ' lr-AdamW/pg2‚1.1349028587418166e-05 trainer/global_step‚18299 % _timestamp‚1.7845234361263194e+09 _runtime‚16609.701942825 _step‚1505á ¯úœÿ4‹9/ _wandbruntime‚16609  _runtime‚16609>å‚Ü4%Œ9…P UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚608343230856704 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329199474 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2805694549921074 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2784473281202896 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _step‚1506 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21253333568000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚629715778420224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚608343212244480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148142165 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21253674992128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148142163 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181047166 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21253674992128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21253333568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2822117772252160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43666281 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329189329 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148142163 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2784473298242256 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2800864438684160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2800864455723520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.784523436127232e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2822118130715648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2805694907521708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43666574 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31923036 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75589610 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚629715648247296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21221268718178 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2800864438684160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31923033 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2822117772252160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 BDeviceStatsMonitor.on_train_batc„Žw^×h_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2800864455723520 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360†!¥Ï(9ŠP N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148142165 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181047166 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329199474 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2805694549921074 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2784473281202896 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 _step‚1506 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21253333568000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚629715778420224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚608343212244480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148142165 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21253674992128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148142163 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181047166 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21253674992128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21253333568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2822117772252160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43666281 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329189329 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148142163 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2784473298242256 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2800864438684160 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2800864455723520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.784523436127232e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2822118130715648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚293 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2805694907521708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43666574 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31923036 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75589610 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚629715648247296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21221268718178 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2800864438684160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31923033 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2822117772252160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2800864455723520 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚608343230856704 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329216937 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148149956 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2784620728909728 _runtime‚16610.312159775 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2801012730225664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21222377106430 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329216937 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21254443666944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚629749943955968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148149958 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75593351 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21222717667816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21373551441408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75593645 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0n2 Y'9•N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31924770 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181056822 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2801012730225664 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181066979 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21373662994944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329206778 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚608376392514560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 % _timestamp‚1.7845234367363577e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2822267156853248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21254785098240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2805843088976798 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2784620728909728 _runtime‚16610.312159775 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2801012730225664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21222377106430 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329216937 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21254443666944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚629749943955968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148149958 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75593351 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21222717667816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21373551441408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75593645 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2805843446577544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚18299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329216937 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148149956 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2829587066461184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21428323527168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21277092492860 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2791844831615856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚631430441909248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2791844814576496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181541961 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚16652.178002338 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2808278180830720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2813121566508082 _step‚1510 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚610002244360704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 330063631 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181541961 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21308902669824 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181531818 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21428216160768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32009733 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75790982 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148531815 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2829587066461184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21428323527168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21277092492860 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2791844831615856 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚631430441909248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2791844814576496 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181541961 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚16652.178002338 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2808278180830720 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2813121566508082 _step‚1510 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚610002244360704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 330063631 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181541961 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21308902669824 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181531818 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21428216160768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32009733 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75790982 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148531815 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2829736809532928 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21310354200064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181551631 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 330081080 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181551631 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21310354200064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21278200881224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32011467 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21429437812224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32011470 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181541474 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75795050 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148539608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148539608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2829736809532928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚631464736471552 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330091239 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75794760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 330081080 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43783293 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148539606 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43783580 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚631464862443008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7845234792139704e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21310012768768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚610035424630784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2829736451062272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 330091239 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148539606 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21310012768768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 trainer/global_step‚18349 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2808426455332864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2808426455332864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚1511Ê…ú%'¢9•N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2829736809532928 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21310354200064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181551631 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 330081080 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181551631 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21310354200064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21278200881224 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32011467 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21429437812224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32011470 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181541474 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75795050 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148539608 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148539608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2829736809532928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚631464736471552 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330091239 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75794760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 330081080 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43783293 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148539606 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43783580 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚631464862443008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7845234792139704e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21310012768768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚610035424630784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2829736451062272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 330091239 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148539606 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21310012768768 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 trainer/global_step‚18349 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2808426455332864 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2808426455332864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚1511 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚610035406018560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21429330452992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496ç +öÌ4£9/ _wandbruntime‚16652  _runtime‚16652˜]¦*¤9Š % _timestamp‚1.7845234792150276e+09 3loss/SRqqt1/assignment_loss‚0.35488403405111113 /loss/FRt1/detection_loss‚0.5110813072885501 /loss/FBt2/detection_loss‚0.5367914107650311 trainer/global_step‚18349 1loss/SRqqt1/detection_loss‚0.6952560103947065 1loss/SRqqt2/detection_loss‚0.6436517256793691 epoch‚45 _runtime‚16652.790407722 _step‚1512 /loss/FRt1/assignment_loss‚1.531036450471596 0loss/FRt2/assignment_loss‚0.7100872283202436 /loss/FRt2/detection_loss‚1.5292461482820303 %loss/total_loss‚7.400189554499477 3loss/SRqqt2/assignment_loss‚0.44405404772383034 2loss/FBt1/assignment_loss‚0.017359945595132254 4loss/FBt2/assignment_loss‚0.00017005687076677566 /loss/FBt1/detection_loss‚0.4265711890571099ß¿ö$¥9 3loss/SRqqt1/assignment_loss‚0.35488403405111113 /loss/FRt1/detection_loss‚0.5110813072885501 /loss/FBt2/detection_loss‚0.5367914107650311 trainer/global_step‚18349 1loss/SRqqt1/detection_loss‚0.6952560103947065 1loss/SRqqt2/detection_loss‚0.6436517256793691 epoch‚45 _runtime‚16652.790407722 _step‚1512 /loss/FRt1/assignment_loss‚1.531036450471596 0loss/FRt2/assignment_loss‚0.7100872283202436 /loss/FRt2/detection_loss‚1.5292461482820303 %loss/total_loss‚7.400189554499477 3loss/SRqqt2/assignment_loss‚0.44405404772383034 2loss/FBt1/assignment_loss‚0.017359945595132254 4loss/FBt2/assignment_loss‚0.00017005687076677566 /loss/FBt1/detection_loss‚0.4265711890571099 % _timestamp‚1.7845234792150276e+09è F8S¿„¦9‚(:ù ãÕöÒ¸¶ÃË network.sent‚ 2.9469639e+07& disk.nvme0n1.out‚104679.9697265625( disk.nvme1c1n1.out‚51153.99365234375 disk.md0.out‚ 728.7421875$ disk.dm-0.out‚138173.16064453125 disk.dm-1.in‚0" disk.dm-1.out‚16135.1142578125 disk.rbd0.in‚0 disk.md0.in‚0% disk./.usageGB‚1168.4385871887207" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.rbd0.out‚0& proc.memory.availableMB‚ 480.171875* proc.memory.percent‚18.468022346496582 disk.nvme0n1.in‚ 309.37109375 disk.dm-0.in‚ 539.2421875$ memory_percent‚97.06926345825195# proc.memory.rssMB‚ 3025.80078125 cpu‚25.698844714442412 proc.cpu.threads‚17 network.recv‚ 4.343864e+06) disk./.usagePercent‚76.10738232787628( disk.nvme0c0n1.out‚104679.9697265625" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚51153.99365234375Ç!Þ§9‚(:“ ãÕöÒ´À¸Ì gpu.0.gpu‚64.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚267.697' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.23233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùH¹Þ‘¨9‚(:† òÕöÒ´‰¿Ë gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.053' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.351 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôà3Ÿ…©9‚(:ú òÕöÒ‹š×Ë proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 729.16015625 disk.dm-1.in‚0# proc.memory.rssMB‚ 3025.80078125$ disk./.usageGB‚1168.436752319336 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0" disk.dm-1.out‚16135.2275390625# memory_percent‚97.0707654953003( proc.memory.availableMB‚ 479.92578125* proc.memory.percent‚18.468022346496582 network.recv‚ 4.349509e+06) disk./.usagePercent‚76.10726281187664 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0n1.out‚104710.62939453125( disk.nvme1c1n1.out‚51672.35986328125& disk.nvme1n1.out‚51672.35986328125 network.sent‚ 2.953315e+07) disk.nvme0c0n1.out‚104710.62939453125 disk.dm-0.in‚ 539.2421875# disk.dm-0.out‚138721.1845703125 cpu‚25.698665571759154­ƒ·4ª9/ _wandbruntime‚16694  _runtime‚16694æ>ˆÌ«9Æ ' lr-AdamW/pg1‚1.0060975851758557e-05 ' lr-AdamW/pg2‚1.0060975851758557e-05 trainer/global_step‚18399 % _timestamp‚1.7845235209898362e+09 _runtime‚16694.565434638 _step‚1513÷æÔwѬ9Ë ' lr-AdamW/pg1‚1.0060975851758557e-05 ' lr-AdamW/pg2‚1.0060975851758557e-05 trainer/global_step‚18399 % _timestamp‚1.7845235209898362e+09 _runtime‚16694.565434638 _step‚1513é —Ñ¨¬4­9/ _wandbruntime‚16694  _runtime‚16694;5Y (®9…P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43896814 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148921465 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚633145376398336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 330948078 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7845235209907956e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2837056719133696 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32096436 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43896530 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75992963 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚633145248322560 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2820548583095090 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2815691888898560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚16694.56625243 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148921463 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182016470 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2837056360670208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2815691905937920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148921465 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148921463 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75993250 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182026613 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21332235144994 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚18399 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 330937933 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2837056719133696 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚611661257864704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚611661239252480 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2820548940695724 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _step‚1514 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_t¸ji;rain_batch_start/num_device_alloc‚1452 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7845235209907956e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2837056719133696 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32096436 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43896530 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75992963 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚633145248322560 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2820548583095090 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2815691888898560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚16694.56625243 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148921463 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182016470 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2837056360670208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2815691905937920 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148921465 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148921463 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75993250 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182026613 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21332235144994 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 trainer/global_step‚18399ê ­XwQ…°9‚(:ú ÖöÒ„†üË* proc.memory.percent‚18.468022346496582 network.recv‚ 4.352806e+06 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0n1.out‚104719.98876953125 network.sent‚ 2.9548915e+07 disk.dm-1.in‚0) disk.nvme0c0n1.out‚104719.98876953125 disk.nvme0n1.in‚ 309.37109375 proc.cpu.threads‚17% disk./.usageGB‚1168.4356575012207 disk.dm-0.in‚ 539.2421875' disk.nvme1c1n1.out‚51674.8349609375% disk.nvme1n1.out‚51674.8349609375$ memory_percent‚97.06947803497314( proc.memory.availableMB‚ 480.13671875# proc.memory.rssMB‚ 3025.80078125 cpu‚25.698486324586796) disk./.usagePercent‚76.10719149983525" disk.dm-0.out‚138732.115234375# disk.dm-1.out‚16135.41455078125" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 729.4921875ÑØõÏ“±9‚(:ˆ ÖöÒœÊÍÌ gpu.0.gpu‚97.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚270.165' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚90.055 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ0è­4²9/ _wandbruntime‚16695  _runtime‚16695vZ?'³9ŽN SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148929258 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚611694438134784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2815840163400704 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2815840163400704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _step‚1515 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148929258 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148929256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2799363795656928 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2799363778617568 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 330965541 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32098167 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 trainer/global_step‚18399 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75996967 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2837205745271296 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182036283 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21365581870592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75997257 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚633179550409216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330965541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 330955382 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182036283 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43899087 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚633179667992064 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 182026126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21333684094632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287 $ _timestamp‚1.784523521603686e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚611694419522560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21333343533246 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2837206103741952 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2815840180440064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43898800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21485229857280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2820697122150814 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21365923301888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21365923301888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148929256 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180œ~ÿ'´9“N $ _timestamp‚1.784523521603686e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚611694419522560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21333343533246 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2837206103741952 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2815840180440064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43898800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21485229857280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2820697122150814 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21365923301888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21365923301888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148929256 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148929258 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚611694438134784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2815840163400704 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2815840163400704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 _step‚1515 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148929258 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148929256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2799363795656928 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2799363778617568 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 330965541 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32098167 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 trainer/global_step‚18399 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75996967 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2837205745271296 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182036283 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21365581870592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75997257 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚633179550409216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330965541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 330955382 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182036283 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43899087 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚633179667992064 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 182026126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21333684094632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚287ë ¹àö4µ9/ _wandbruntime‚16695  _runtime‚16695@u£b¶9ˆ trainer/global_step‚18399 _step‚1516 0loss/FRt2/assignment_loss‚0.6600370596094829 0loss/SRqqt1/detection_loss‚0.676442627018724 1loss/SRqqt2/detection_loss‚0.6026376115341034 epoch‚45 $ _timestamp‚1.784523521604615e+09 _runtime‚16695.179926097 0loss/FRt1/assignment_loss‚1.5475852218593737 2loss/SRqqt1/assignment_loss‚0.3822029671323956 4loss/FBt1/assignment_loss‚0.00029570552880986065 3loss/FBt2/assignment_loss‚6.937523726757194e-05 /loss/FRt1/detection_loss‚0.4611552717128459 /loss/FBt1/detection_loss‚0.3711536431631342 3loss/SRqqt2/assignment_loss‚0.46423947072301897 .loss/FRt2/detection_loss‚1.534113325552288 /loss/FBt2/detection_loss‚0.4087608776829912 %loss/total_loss‚7.108693156754436Zœê€·9 /loss/FBt1/detection_loss‚0.3711536431631342 3loss/SRqqt2/assignment_loss‚0.46423947072301897 .loss/FRt2/detection_loss‚1.534113325552288 /loss/FBt2/detection_loss‚0.4087608776829912 %loss/total_loss‚7.108693156754436 trainer/global_step‚18399 _step‚1516 0loss/FRt2/assignment_loss‚0.6600370596094829 0loss/SRqqt1/detection_loss‚0.676442627018724 1loss/SRqqt2/detection_loss‚0.6026376115341034 epoch‚45 $ _timestamp‚1.784523521604615e+09 _runtime‚16695.179926097 0loss/FRt1/assignment_loss‚1.5475852218593737 2loss/SRqqt1/assignment_loss‚0.3822029671323956 4loss/FBt1/assignment_loss‚0.00029570552880986065 3loss/FBt2/assignment_loss‚6.937523726757194e-05 /loss/FRt1/detection_loss‚0.4611552717128459ì ïÄÕ@‚¸9‚(:÷ ÖöÒƒˆåË disk.rbd0.in‚0' disk.nvme0c0n1.out‚104721.525390625% disk.nvme0n1.out‚104721.525390625 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 729.8828125 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme1c1n1.out‚51675.4462890625 disk.md0.in‚0# memory_percent‚97.0710039138794 cpu‚25.700610672104602 proc.cpu.threads‚17 network.recv‚ 4.357727e+06% disk./.usageGB‚1168.4357566833496$ disk.dm-0.out‚138733.35302734375" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375% disk.nvme1n1.out‚51675.4462890625( proc.memory.availableMB‚ 479.88671875* proc.memory.percent‚18.468022346496582 network.sent‚ 2.9600694e+07) disk./.usagePercent‚76.10719796015955 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚16135.50048828125 disk.nvme0n1.in‚ 309.37109375# proc.memory.rssMB‚ 3025.80078125l ¡¹9‚(:’ ÖöÒõåË gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.758' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.919333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_´4º9/ _wandbruntime‚16711  _runtime‚16711\ˆ‡I »9Ëz 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6702240705490112 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11931981146335602 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2851065397262573 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4810965359210968 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6553584933280945 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.695489227771759 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29474711418151855 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.29488256573677063 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6555539965629578 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6637672781944275 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.060345251113176346 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8151738047599792 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.060345251113176346 *jet/accuracy_1_of_3‚0.9195078015327454 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6917639970779419 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3457472324371338 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27013200521469116 +jet/accuracy_3_of_4‚0.08844424784183502 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2545073330402374 *jet/accuracy_1_of_1‚0.3327461779117584 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5806400179862976 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.629554271697998 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9976974725723267 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6279866695404053 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5827351212501526 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.684593677520752 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FR™Ëç0t_purity‚0.2732834815979004 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7106047868728638 +jet/accuracy_2_of_2‚0.16198024153709412 particle/accuracy_1_of_2‚1 % _timestamp‚1.7845235376109784e+09 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26206985116004944 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6679377555847168 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 trainer/global_step‚18399 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7541402578353882 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3001895546913147 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 /particle/accuracy_4_of_4‚0.9539412260055542 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4845624566078186 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975182414054871 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5827351212501526 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8403250575065613 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29774171113967896 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21150967478752136 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3185391128063202 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 +particle/specificity‚0.9104509949684143 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.780332088470459 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5729631185531616 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4849252700805664 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2284117490053177 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6171715259552002 particle/accuracy_2_of_2‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21866381168365479 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5834149122238159 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 particle/accuracy_4_of_5‚1 )loss/val_total_loss‚7.421728968808576 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.833541214466095 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 _runtime‚16711.18711247 _step‚1517 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28124356269836426 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35171613097190857 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.780332088470459 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 +particle/sensitivity‚0.6442778706550598 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7591648101806641 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07310578972101212 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0512702539563179 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8171957731246948 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07385163754224777 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7377762198448181 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.774107813835144 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 )jet/accuracy_2_of_3‚0.562374472618103 7validation_average_jet_accuracy‚0.37368130683898926 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6297198534011841 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6024929881095886 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6297288537025452 *jet/accuracy_2_of_4‚0.7257912755012512 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8838788866996765 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2257821410894394 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9923719763755798 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.632777750492096 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21341359615325928 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2736569941043854 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26296576857566833 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06142380088567734 +jet/accuracy_3_of_3‚0.07468684762716293 jet/accuracy_1_of_5‚1 epoch‚45 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7187678217887878 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5045710802078247 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6035434603691101 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45159363746643066 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6019079685211182 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4707694351673126 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3403964638710022 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7206034660339355 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.584128201007843 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7151849269866943 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6370507478713989 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3003405034542084 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2894572615623474 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21553963422775269 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8682984709739685 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 /particle/accuracy_3_of_3‚0.9994120597839355 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34738120436668396 (particle/accuracy‚0.8532869219779968 particle/accuracy_3_of_5‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9456977844238281 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2234165072441101 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06941928714513779 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23565860092639923 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06988149136304855 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.909145712852478 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6057165861129761 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6476525068283081 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7331722974777222 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6387386322021484 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2902858555316925 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3046208918094635 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5796656608581543 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5722929835319519 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04062904044985771 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09921771287918091 'particle/f_score‚0.6535048484802246 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.301983118057251 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2894572615623474 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.646133303642273 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5708333253860474 particle/accuracy_2_of_3‚1 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7087320685386658 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6560360193252563 *jet/accuracy_1_of_4‚0.9695299863815308 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9920614957809448 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt*FBt/event_purity‚0.051535714417696 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.833541214466095FÇ©ßV=¼9Ðz =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2736569941043854 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26296576857566833 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.06142380088567734 +jet/accuracy_3_of_3‚0.07468684762716293 jet/accuracy_1_of_5‚1 epoch‚45 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7187678217887878 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5045710802078247 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6035434603691101 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45159363746643066 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6019079685211182 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4707694351673126 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3403964638710022 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7206034660339355 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.584128201007843 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7151849269866943 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6370507478713989 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3003405034542084 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2894572615623474 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964779615402222 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21553963422775269 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8682984709739685 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 /particle/accuracy_3_of_3‚0.9994120597839355 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34738120436668396 (particle/accuracy‚0.8532869219779968 particle/accuracy_3_of_5‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9456977844238281 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2234165072441101 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06941928714513779 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23565860092639923 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06988149136304855 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.909145712852478 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6057165861129761 particle/accuracy_1_of_1‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6476525068283081 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7331722974777222 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6387386322021484 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2902858555316925 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3046208918094635 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5796656608581543 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5722929835319519 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04062904044985771 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09921771287918091 'particle/f_score‚0.6535048484802246 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.301983118057251 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2894572615623474 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.646133303642273 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5708333253860474 particle/accuracy_2_of_3‚1 jet/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7087320685386658 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6560360193252563 *jet/accuracy_1_of_4‚0.9695299863815308 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9920614957809448 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt*FBt/event_purity‚0.051535714417696 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.833541214466095 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6702240705490112 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11931981146335602 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2851065397262573 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4810965359210968 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6553584933280945 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.695489227771759 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29474711418151855 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.29488256573677063 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6555539965629578 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6637672781944275 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.060345251113176346 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8151738047599792 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.060345251113176346 *jet/accuracy_1_of_3‚0.9195078015327454 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6917639970779419 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3457472324371338 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27013200521469116 +jet/accuracy_3_of_4‚0.08844424784183502 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2545073330402374 *jet/accuracy_1_of_1‚0.3327461779117584 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5806400179862976 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.629554271697998 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9976974725723267 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6279866695404053 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5827351212501526 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.684593677520752 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2732834815979004 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7106047868728638 +jet/accuracy_2_of_2‚0.16198024153709412 particle/accuracy_1_of_2‚1 % _timestamp‚1.7845235376109784e+09 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26206985116004944 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6679377555847168 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 trainer/global_step‚18399 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7541402578353882 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3001895546913147 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 /particle/accuracy_4_of_4‚0.9539412260055542 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4845624566078186 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9975182414054871 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5827351212501526 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8403250575065613 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29774171113967896 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21150967478752136 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3185391128063202 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 +particle/specificity‚0.9104509949684143 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.780332088470459 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5729631185531616 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4849252700805664 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2284117490053177 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6171715259552002 particle/accuracy_2_of_2‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21866381168365479 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5834149122238159 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 particle/accuracy_4_of_5‚1 )loss/val_total_loss‚7.421728968808576 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.833541214466095 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 _runtime‚16711.18711247 _step‚1517 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28124356269836426 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35171613097190857 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.780332088470459 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 +particle/sensitivity‚0.6442778706550598 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7591648101806641 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07310578972101212 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0512702539563179 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8171957731246948 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07385163754224777 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7377762198448181 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.774107813835144 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 )jet/accuracy_2_of_3‚0.562374472618103 7validation_average_jet_accuracy‚0.37368130683898926 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6297198534011841 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6024929881095886 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6297288537025452 *jet/accuracy_2_of_4‚0.7257912755012512 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8838788866996765 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2257821410894394 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9923719763755798 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.632777750492096 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21341359615325928 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1í )9î|½9‚(:ñ ŸÖöÒ–âáË) disk./.usagePercent‚76.09170162994181 disk.md0.out‚730.3125 disk.dm-0.in‚ 539.2421875 disk.dm-0.out‚ 138737.390625 disk.dm-1.in‚0& disk.nvme0n1.out‚104725.7939453125 disk.md0.in‚0 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚51676.25927734375$ memory_percent‚97.06964492797852# proc.memory.rssMB‚ 3025.80078125 cpu‚25.70064007670445" disk.nvme0c0n1.in‚ 309.37109375 network.recv‚ 4.362646e+06% disk./.usageGB‚1168.1978492736816( disk.nvme1c1n1.out‚51676.25927734375& proc.memory.availableMB‚ 480.109375* proc.memory.percent‚18.468022346496582# disk.dm-1.out‚16135.61376953125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚104725.7939453125 proc.cpu.threads‚17 disk.rbd0.out‚0 network.sent‚ 2.9652849e+07ÿd@ß¾9‚(:” ŸÖöÒÊŸâË gpu.0.gpu‚90.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚123.498' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.166000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Eÿ9‚(:ô ®ÖöÒˆØ“Ì network.sent‚ 2.9698843e+07) disk./.usagePercent‚76.09171057500623 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375) disk.nvme0c0n1.out‚104726.78955078125 disk.nvme0n1.in‚ 309.37109375' disk.nvme1c1n1.out‚51676.8681640625% disk.nvme1n1.out‚51676.8681640625 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚16135.68408203125" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 730.78125$ memory_percent‚97.07117080688477# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 disk.rbd0.out‚0 disk.nvme1n1.in‚ 229.87109375& proc.memory.availableMB‚ 479.859375 network.recv‚ 4.36774e+06% disk./.usageGB‚1168.1979866027832! disk.dm-0.out‚138737.93359375 disk.rbd0.in‚0' disk.nvme0n1.out‚104726.78955078125 cpu‚25.700533811736214 proc.cpu.threads‚17lGÂë“À9‚(:ˆ ®ÖöÒ›ÓðË gpu.0.gpu‚80.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚272.805' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚90.935 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qî-z4Á9/ _wandbruntime‚16754  _runtime‚16754kËÚŸÈÂ9 & lr-AdamW/pg1‚9.445560441416122e-06 & lr-AdamW/pg2‚9.445560441416122e-06 trainer/global_step‚18449 % _timestamp‚1.7845235810580254e+09 _runtime‚ 16754.6336638 _step‚1518dVÍÃ9Ç & lr-AdamW/pg1‚9.445560441416122e-06 & lr-AdamW/pg2‚9.445560441416122e-06 trainer/global_step‚18449 % _timestamp‚1.7845235810580254e+09 _runtime‚ 16754.6336638 _step‚1518î ,ß .4Ä9/ _wandbruntime‚16754  _runtime‚16754zâå§+Å9…P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 182582616 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2846488675054080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚634903419795456 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2808509137331920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2846488316590592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21560685257216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2846488675054080 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21440700133376 XJDeviceStatsMonitor.on_train_batch_start/aBâ"à!llocated_bytes.small_pool.current‚ 341424128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _step‚1519 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21440358709248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32187252 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21560588376576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332016537 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2825047974920704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 182592759 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44045122 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21408362768573 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚16754.635742804 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149433921 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2829917159539219 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76232648 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76232371 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2846488316590592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21408022207299 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2825047957881344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182582616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2829917517139853 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2808509154371280 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149433921 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21440700133376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149433923 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚613342831418880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149433923 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚613342850031104 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332026682 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76232648 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76232371 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2846488316590592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21408022207299 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2825047957881344 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182582616 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2829917517139853 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2808509154371280 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149433921 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21440700133376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149433923 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚613342831418880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149433923 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚613342850031104 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332026682 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2808509137331920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2846488316590592 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21560685257216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2846488675054080 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21440700133376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 _step‚1519 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21440358709248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32187252 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21560588376576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332016537 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2825047974920704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 182592759 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44045122 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21408362768573 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚16754.635742804 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149433921 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2829917159539219 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344ï V’€Ç9‚(:õ ½ÖöÒ®¹¿Ë disk.dm-1.in‚0# disk.dm-1.out‚16135.82763671875 network.sent‚ 2.9701866e+07) disk./.usagePercent‚76.09171107195426" disk.nvme1c1n1.in‚ 229.87109375# disk.nvme1n1.out‚51679.07421875 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 4.370858e+06% disk.nvme1c1n1.out‚51679.07421875" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375$ memory_percent‚97.07376956939697# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚138746.50634765625( disk.nvme0c0n1.out‚104734.4091796875& disk.nvme0n1.out‚104734.4091796875 cpu‚25.70028204131272% disk./.usageGB‚1168.1979942321777 disk.md0.out‚ 731.3046875( proc.memory.availableMB‚ 479.43359375dØN“È9‚(:ˆ ½ÖöÒ¢ê¸Ì gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚276.447' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.149 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤Þ²u4É9/ _wandbruntime‚16755  _runtime‚16755Ó©ú'Ê9N QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76236395 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32188986 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32188983 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21561703270912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182602429 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2825196249422848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21409471156937 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2846638059662336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚613376011688960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21441468808192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332044145 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21441468808192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚18449 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 332033986 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2830066056195689 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21409130595551 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845235816849833e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2830065698594943 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _runtime‚16755.260757733 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332044145 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149441716J¶¯i‚ Ë9•N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 182602429 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2846637701191680 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149441716 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21441810239488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚634937714959872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2808656567999392 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76236678 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76236395 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32188986 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32188983 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21561703270912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182602429 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2825196249422848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21409471156937 E?DeviceStatsMonitor.on_train_batch_eWrM£™nd/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2846638059662336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚613376011688960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21441468808192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332044145 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21441468808192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚18449 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 332033986 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2830066056195689 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21409130595551 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 % _timestamp‚1.7845235816849833e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2830065698594943 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _runtime‚16755.260757733 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332044145 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149441716 QDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2853957610799616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21463845981981 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44160680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚636618508527104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21496269235200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2832461682988544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332900984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2853957969263104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21495927811072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚615001863535104 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2832461682988544 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76434629 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149823571 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2832461700027904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2837344533726861 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2815880670705520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21496269235200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21616539722752 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149823571 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚615001844922880 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2837344176126227 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183067268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32273952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21616644992000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21463505420707 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2832461700027904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149823573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21495927811072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183077411 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2853957610799616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21463845981981 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44160680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚636618508527104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21496269235200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2832461682988544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332900984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2853957969263104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21495927811072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚615001863535104 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2832461682988544 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76434629 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149823571 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2832461700027904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2837344533726861 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2815880670705520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21496269235200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21616539722752 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149823571 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚615001844922880 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2837344176126227 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183067268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32273952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21616644992000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21463505420707 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2832461700027904 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149823573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21495927811072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183077411 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332918447 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183076924 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76438892 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2816028101372992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21464613808959 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332918447 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76438605 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2816028118412352 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183076924 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2854107353871360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚615035043805184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2854107353871360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚636652680067584 _step‚1524 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2837493072782697 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332908288 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 332908288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚18499 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183087081 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚16800.368264196 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21617762233856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149831366ª–²‹'Ý9•N FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76438605 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2816028118412352 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183076924 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2854107353871360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚615035043805184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2854107353871360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚636652680067584 _step‚1524 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2837493072782697 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332908288 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 332908288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚18499 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183087081 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚16800.368264196 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21617762233856 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149831366 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332918447 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183076924 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76438892 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2816028101372992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21464613808959 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332918447 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21551838337024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21519329195389 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150213223 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 333765141 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183562063 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2861427263472128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76636595 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚1527 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2861427263472128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150213221 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76636308 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2844771192713235 trainer/global_step‚18549 eQDeviceStatsMonitor.on_train_batch_s1}c[ctart/inactive_split_bytes.large_pool.allocated‚616660877039104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21551496912896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2861426905008640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183562063 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183551920 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2861426905008640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44275659 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2823252221118480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21551838337024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚16845.132190868 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2839875408095744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44275943 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚616660858426880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32360652 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21551496912896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2844771550313869 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150213221 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32360649 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚638333250172928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 % _timestamp‚1.7845236715566316e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21551838337024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21519329195389 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150213223 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 333765141 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 183562063 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2861427263472128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76636595 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚1527 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2861427263472128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150213221 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76636308 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2844771192713235 trainer/global_step‚18549 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚616660877039104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21551496912896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2861426905008640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183562063 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183551920 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2861426905008640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44275659 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2823252221118480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21551838337024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚16845.132190868 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2839875408095744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44275943 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚616660858426880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32360652 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21551496912896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2844771550313869 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150213221 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32360649 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚638333250172928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 % _timestamp‚1.7845236715566316e+09 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚638367550534144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150221014 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76640360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2844919731768959 _runtime‚16845.762487079 _step‚1528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2823399651785952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2840023682597888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21673612904960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21552607011840 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150221014 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183571733 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚638367670214144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2840023682597888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183571733 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2840023699637248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76640651 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183561576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚616694038696960 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2861576289609728 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2844920089369705 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 333792749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21520097022367 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528‡P±Ü ï9”N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183571733 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2840023699637248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76640651 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚288 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183561576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚616694038696960 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2861576289609728 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2844920089369705 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 333792749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21520097022367 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21520437583753 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 333782590 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21552948443136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚638367550534144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150221014 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76640360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2844919731768959 _runtime‚16845.762487079 _step‚1528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2823399651785952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2840023682597888 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21673612904960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21552607011840 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150221014 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183571733 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚638367670214144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2840023682597888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180ø Óe½4ð9/ _wandbruntime‚16845  _runtime‚16845åñ9‰ 1loss/SRqqt2/detection_loss‚0.6141134562378765 0loss/FBt1/detection_loss‚0.46813754406617797 /loss/FBt2/detection_loss‚0.5343080462705807 0loss/FRt2/assignment_loss‚0.5758527348290629 2loss/SRqqt2/assignment_loss‚0.4445492918923708 3loss/FBt2/assignment_loss‚2.691196824578239e-05 trainer/global_step‚18549 _step‚1529 0loss/FRt1/assignment_loss‚1.4577499352201653 2loss/SRqqt1/assignment_loss‚0.3440220500456024 4loss/FBt1/assignment_loss‚0.00016737276204794192 /loss/FRt1/detection_loss‚0.4173735567219342 1loss/SRqqt1/detection_loss‚0.4506651457187902 % _timestamp‚1.7845236721878483e+09 _runtime‚16845.76316278 /loss/FRt2/detection_loss‚1.4700942898258353 $loss/total_loss‚6.77706033555869 epoch‚46ùk]½ò9Ž % _timestamp‚1.7845236721878483e+09 _runtime‚16845.76316278 /loss/FRt2/detection_loss‚1.4700942898258353 $loss/total_loss‚6.77706033555869 epoch‚46 1loss/SRqqt2/detection_loss‚0.6141134562378765 0loss/FBt1/detection_loss‚0.46813754406617797 /loss/FBt2/detection_loss‚0.5343080462705807 0loss/FRt2/assignment_loss‚0.5758527348290629 2loss/SRqqt2/assignment_loss‚0.4445492918923708 3loss/FBt2/assignment_loss‚2.691196824578239e-05 trainer/global_step‚18549 _step‚1529 0loss/FRt1/assignment_loss‚1.4577499352201653 2loss/SRqqt1/assignment_loss‚0.3440220500456024 4loss/FBt1/assignment_loss‚0.00016737276204794192 /loss/FRt1/detection_loss‚0.4173735567219342 1loss/SRqqt1/detection_loss‚0.4506651457187902ù ªÈ ‚ó9‚(:÷ ¦×öÒÅŒÍË* proc.memory.percent‚18.468022346496582) disk./.usagePercent‚76.10718503951095 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.rbd0.out‚0& disk.nvme0c0n1.out‚104942.65234375( proc.memory.availableMB‚ 479.12109375% disk./.usageGB‚1168.4355583190918( disk.nvme1c1n1.out‚51738.51416015625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 733.94140625# proc.memory.rssMB‚ 3025.80078125$ disk.nvme0n1.out‚104942.65234375& disk.nvme1n1.out‚51738.51416015625 disk.dm-0.in‚ 539.2421875" disk.dm-1.out‚16136.5263671875 disk.rbd0.in‚0 cpu‚25.700205874927107 proc.cpu.threads‚17 network.sent‚ 2.9938389e+07 network.recv‚ 4.40149e+06$ disk.dm-0.out‚139007.82080078125 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375$ memory_percent‚97.07567691802979³ÑÊžô9‚(:“ ¦×öÒ—ÜÌ gpu.0.gpu‚67.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚285.809' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.26966666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†RPÓ~õ9‚(:ó µ×öÒŒäîË disk.nvme1n1.in‚ 229.87109375# memory_percent‚97.0754861831665) disk./.usagePercent‚76.10718106392676 disk.rbd0.out‚0( disk.nvme1c1n1.out‚51739.55712890625( proc.memory.availableMB‚ 479.15234375 proc.cpu.threads‚17 disk.md0.out‚ 734.40625 disk.dm-0.out‚ 139011.03125& disk.nvme0n1.out‚104945.9658203125" disk.nvme1c1n1.in‚ 229.87109375* proc.memory.percent‚18.468022346496582 cpu‚25.700308454827283 network.sent‚ 2.9988838e+07 network.recv‚ 4.406367e+06# disk.dm-1.out‚16136.68505859375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375& disk.nvme1n1.out‚51739.55712890625# proc.memory.rssMB‚ 3025.80078125% disk./.usageGB‚1168.4354972839355 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚104945.9658203125A!—&žö9‚(:“ µ×öÒ‘•±Ì gpu.0.gpu‚93.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚282.371' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.12366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 mSªz÷9‚(:ï Ä×öÒäÕÚË" disk.nvme1n1.out‚ 51739.9140625 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0 disk.nvme0n1.in‚ 309.37109375$ disk.nvme1c1n1.out‚ 51739.9140625 disk.md0.out‚ 734.59375( proc.memory.availableMB‚ 479.14453125# proc.memory.rssMB‚ 3025.80078125 network.sent‚ 2.9993676e+07) disk./.usagePercent‚76.10717733681658% disk./.usageGB‚1168.4354400634766 disk.md0.in‚0! disk.dm-0.out‚139011.66015625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0n1.out‚104946.71337890625# disk.dm-1.out‚16136.73583984375 disk.rbd0.out‚0) disk.nvme0c0n1.out‚104946.71337890625" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.07553386688232* proc.memory.percent‚18.468022346496582 cpu‚25.7002495537382 proc.cpu.threads‚17 network.recv‚ 4.409821e+06 disk.nvme1n1.in‚ 229.87109375%[µNœø9‚(:‘ Ä×öÒýܪÌ gpu.0.gpu‚8.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.721' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.57366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÉì‚4ù9/ _wandbruntime‚16890  _runtime‚16890áĈÊú9Ä _step‚1530 & lr-AdamW/pg1‚7.714228671092147e-06 & lr-AdamW/pg2‚7.714228671092147e-06 trainer/global_step‚18599 % _timestamp‚1.7845237169366314e+09 _runtime‚16890.512318852>G¸µÏû9É & lr-AdamW/pg1‚7.714228671092147e-06 & lr-AdamW/pg2‚7.714228671092147e-06 trainer/global_step‚18599 % _timestamp‚1.7845237169366314e+09 _runtime‚16890.512318852 _step‚1530ú çøËÖ4ü9/ _wandbruntime‚16890  _runtime‚16890[ÌN (ý9†P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32447349 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21728340512768 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21574812408797 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184036572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2830623737452720 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21607407438848 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150602873 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2868896557681152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150602871 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21607066014720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76837572 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21728235243520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334639443 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184046715 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚618319890543104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2852198566900877 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21574471847523 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2847289150242304 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2830623754492080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2868896199217664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2868896557681152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334649588 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76837290 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚18599 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚640048107174400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2868896199217664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2847289150242304 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚16890.513478423 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184046715 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21607407438848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150602871 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845237169378579e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21607066014720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚640048231055872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚618319871930880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32447352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2847289133202944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184036572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44389941 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44390220 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0Rdí,(þ9‹P RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21574812408797 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184036572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2830623737452720 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21607407438848 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150602873 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2868896557681152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150602871 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21607066014720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76837572 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21728235243520 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334639443 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184046715 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚618319890543104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2852198566900877 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21574471847523 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2847289150242304 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2830623754492080 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2868896199217664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2868896557681152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334649588 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76837290 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 trainer/global_step‚18599 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚640048107174400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2868896199217664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2847289150242304 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 _runtime‚16890.513478423 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184046715 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21607407438848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150602871 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845237169378579e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21607066014720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚640048231055872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚618319871930880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32447352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2847289133202944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184036572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44389941 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44390220 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32447349 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21728340512768 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0û Y$»4ÿ9/ _wandbruntime‚16891  _runtime‚168916ßÓ« €:N E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚618353070813184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21729456657408 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150610666 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚640082527470592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2830771185159552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184046228 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32449083 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150610664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2847437424744448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2830771168120192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21575580235775 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76841615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184056385 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2847437424744448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2852347105956713 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D6P:êíˆDeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334656892 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21575920797161 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16891.142580656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44392529 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _step‚1532 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150610664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.784523717566751e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21608176113664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21729349298176 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76841326 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334667051 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21608517544960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334667051 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184046228 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2847437407705088 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2852346748355967 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180}B}N':”N VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32449086 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2869045942289408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21608176113664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44392243 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚18599 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334667051 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21608517544960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334667051 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184046228 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2847437407705088 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2852346748355967 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚618353070813184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21729456657408 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150610666 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚640082527470592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2830771185159552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184046228 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32449083 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150610664 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2847437424744448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2830771168120192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21575580235775 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚286 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76841615 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184056385 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2847437424744448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2852347105956713 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334656892 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21575920797161 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16891.142580656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44392529 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _step‚1532 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150610664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.784523717566751e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21608176113664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21729349298176 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76841326 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2876365493426688 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2876365493426688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21662635116544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚619978885434880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21784151532544 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2859625583487885 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚641763055579648 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2859625225887251 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2876365851890176 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21662635116544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚1535 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2854702858310144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 335523890 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184531367 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2854702875349504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184521224 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21662976540672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚619978904047104 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2854702875349504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77038415 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32534049 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21629955060931 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184521224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21784046263296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚18649 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21662976540672 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150992523 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2854702858310144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21630295622205 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150992523 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184531367 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2837995270826320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150992523 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184531367 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2837995270826320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2876365493426688 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2876365493426688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21662635116544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚619978885434880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21784151532544 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2859625583487885 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚641763055579648 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2859625225887251 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2876365851890176 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21662635116544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚1535 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2854702858310144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 335523890 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184531367 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2854702875349504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184521224 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21662976540672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚619978904047104 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2854702875349504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚296 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77038415 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32534049 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21629955060931 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184521224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21784046263296 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚18649 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21662976540672 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150992523 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2854702858310144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21630295622205 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77042215 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21663745215488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620012084317184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚1536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21785163643904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184530880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44506432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32535783 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151000316 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21785266808832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151000314 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184541037 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21664086646784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚641797229348864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620012065704960 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2838142718533152 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2854851149851648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2854851132812288 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2876515236498432 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 335531194 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 335541353 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77042215 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21663745215488 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620012084317184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚1536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21785163643904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184530880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44506432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32535783 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151000316 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21785266808832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151000314 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184541037 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21664086646784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚641797229348864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620012065704960 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2838142718533152 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2854851149851648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2854851132812288 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2876515236498432 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 335531194 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 335541353 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2883834787635712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21718204218368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151382173 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21718545642496 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2883834787635712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21718204218368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44619030 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21685778835613 _runtime‚16980.276100151 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2862116600456704 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2883835146099200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44618750 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151382171 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77239499 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2862116600456704 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2862116583417344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151382171 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚643477802207744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2845366821239280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚643477921894912 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚18699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185005876 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185016019 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 % _timestamp‚1.7845238067004778e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151382171 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77239499 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2862116600456704 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2862116583417344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151382171 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚643477802207744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2845366821239280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚643477921894912 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 trainer/global_step‚18699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185005876 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185016019 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 % _timestamp‚1.7845238067004778e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2883834787635712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚21718204218368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ƒ 'T&«4£:/ _wandbruntime‚16980  _runtime‚16980½bmp&¤:N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185015532 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2867200781529983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2845514234867392 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336405496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚643512217424384 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21719314317312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151389966 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336415655 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21686887223977 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _runtime‚16980.905588777 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77243776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77243495 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2845514251906752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151389964 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2862264874958848 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2862264857919488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚621671079208960 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2867201139130729 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336415655 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44621290 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚621671097821184 trainer/global_step‚18699 _step‚1540 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32622486 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44621012 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [FDeviceÙ¡ßïStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2862264857919488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21841022729728 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2883984530707456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21686546662591 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151389966 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185025689 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 % _timestamp‚1.7845238073298006e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21686887223977 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _runtime‚16980.905588777 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77243776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77243495 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2845514251906752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151389964 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2862264874958848 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2862264857919488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚621671079208960 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2867201139130729 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336415655 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44621290 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚621671097821184 trainer/global_step‚18699 _step‚1540 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32622486 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44621012 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2862264857919488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21841022729728 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2883984530707456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21686546662591 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151389966 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185025689 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 % _timestamp‚1.7845238073298006e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2845514234867392 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336405496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚643512217424384 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21719314317312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151389966 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336415655 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185500671 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2874479259061267 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2891304081844736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151771823 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21895835794944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151771821 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2869530325563904 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2891304440308224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32707452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44733729 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2869530308524544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2891304081844736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21773773320192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151771823 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185500671 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚645192766850048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151771821 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77440879 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7845238513767893e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77441181 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2869530325563904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21895720039936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185490528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44733430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚623296931055104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21740921487747 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534ãÙjœñ²:‹P UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337272494 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185500671 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2874479259061267 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2891304081844736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151771823 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21895835794944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151771821 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2869530325563904 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2891304440308224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32707452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44733729 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2869530308524544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2891304081844736 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21773773320192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 Ì»l ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151771823 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185500671 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚645192766850048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151771821 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77440879 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7845238513767893e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77441181 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2869530325563904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21895720039936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185490528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44733430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚623296931055104 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21740921487747 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21741262049021 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185490528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360‡ =·€³:‚(:õ ËØöÒ³î†Ì# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 cpu‚25.698471356539596 network.recv‚ 4.444489e+06( disk.nvme1c1n1.out‚53782.21044921875& disk.nvme1n1.out‚53782.21044921875% proc.memory.availableMB‚ 479.40625 network.sent‚ 3.0220523e+07 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 proc.cpu.threads‚17% disk./.usageGB‚1168.4423179626465# disk.dm-0.out‚141783.8017578125 disk.dm-1.out‚ 16137.796875 disk.rbd0.out‚0) disk.nvme0c0n1.out‚105684.77783203125 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 737.91015625( disk./.usagePercent‚76.1076253354598 disk.rbd0.in‚0' disk.nvme0n1.out‚105684.77783203125 disk.md0.in‚0$ memory_percent‚97.07393646240234Ý }œ‘´:‚(:† ËØöÒÑÁÅÌ gpu.0.gpu‚3.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.051' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.017 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê’ðR4µ:/ _wandbruntime‚17025  _runtime‚17025ê5è'¶:N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21775224850432 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151779616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185510341 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2869678600066048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337289957 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2891453466445824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2891453824916480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77445154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚645227062945280 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2874627798116991 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚18749 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21774883419136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21774883419136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2874628155717737 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337279798 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77444872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32709183 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚645226926488064 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2852885785280352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185500184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21742370437385 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚623330092712960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21896951620096 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44735968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44735689 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151779616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2869678583026688 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21775224850432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0zÁµú'·:•N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185510341 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _runtime‚17025.567863045 _step‚1544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32709186 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 % _timestamp‚1.7845238519920812e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337289957 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77444872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32709183 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚645226926488064 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2852885785280352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185500184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21742370437385 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚623330092712960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21896951620096 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44735968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44735689 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151779616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2869678583026688 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21775224850432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21775224850432 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151779616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185510341 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2869678600066048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337289957 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2891453466445824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2891453824916480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77445154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚645227062945280 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2874627798116991 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚18749 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21774883419136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21774883419136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2874628155717737 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337279798 E4‚»:‚(:÷ ÚØöÒ ÐùË( disk.nvme1c1n1.out‚53788.95556640625& disk.nvme1n1.out‚53788.95556640625 disk.md0.in‚0# disk.dm-0.out‚141820.8447265625 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375 network.sent‚ 3.0272263e+07 network.recv‚ 4.449096e+06 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875 disk.dm-1.out‚16137.86328125 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0n1.out‚105715.77294921875$ disk./.usageGB‚1168.434642791748" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 738.203125 disk.nvme0n1.in‚ 309.37109375$ memory_percent‚97.07236289978027 cpu‚25.698223849144345 proc.cpu.threads‚17) disk./.usagePercent‚76.10712540574812) disk.nvme0c0n1.out‚105715.77294921875' proc.memory.availableMB‚ 479.6640625# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582~†ž¼:‚(:“ ÚØöÒ‚“ºÌ gpu.0.gpu‚96.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚280.308' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.43599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Ñ4„½:‚(:ù éØöÒ”¾óË disk.md0.out‚ 738.6484375" memory_percent‚97.074294090271* proc.memory.percent‚18.468022346496582 disk.nvme1n1.in‚ 229.87109375# disk.dm-0.out‚141825.1767578125( proc.memory.availableMB‚ 479.34765625# proc.memory.rssMB‚ 3025.80078125 proc.cpu.threads‚17" disk.nvme1c1n1.in‚ 229.87109375 network.recv‚ 4.453924e+06% disk./.usageGB‚1168.4345893859863 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375 disk.dm-0.in‚ 539.2421875) disk./.usagePercent‚76.10712192711195' disk.nvme0n1.out‚105719.86376953125 disk.md0.in‚0 disk.dm-1.in‚0# disk.dm-1.out‚16137.94091796875 cpu‚25.697947318749495' disk.nvme1c1n1.out‚53790.2138671875% disk.nvme1n1.out‚53790.2138671875 network.sent‚ 3.0313498e+07 disk.rbd0.in‚0) disk.nvme0c0n1.out‚105719.86376953125®ûŒ‘¾:‚(:† éØöÒòó¿Ì gpu.0.gpu‚50.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚256.26' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚85.42 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºT4¿:/ _wandbruntime‚17068  _runtime‚17068°ÊÀ:Ä _step‚1546 & lr-AdamW/pg1‚5.675269308804019e-06 & lr-AdamW/pg2‚5.675269308804019e-06 trainer/global_step‚18799 % _timestamp‚1.7845238944304197e+09 _runtime‚17068.005936378 ¹¼ªÏÁ:É & lr-AdamW/pg2‚5.675269308804019e-06 trainer/global_step‚18799 % _timestamp‚1.7845238944304197e+09 _runtime‚17068.005936378 _step‚1546 & lr-AdamW/pg1‚5.675269308804019e-06Š ÛãºÎ4Â:/ _wandbruntime‚17068  _runtime‚17068l<ö]Ã:†P MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21796745262429 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152161471 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 338136651 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2898773376053760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2876944033631744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185985323 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338146796 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44848095 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2881906633248909 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21829683846144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2860109870947120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44848377 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21951519016960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2876944033631744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚17068.006672789 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32794152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77642529 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77642244 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2860109887986480 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152161473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32794149 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21829342422016 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21951630577664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 % _timestamp‚1.7845238944312365e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21829683846144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152161471 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚646907444963840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185975180 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚18799 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚624955925946880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2881906275648275 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2898773734517248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚624955944559104 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274§,Ê(Ä:‹P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152161471 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚646907444963840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185975180 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚18799 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚624955925946880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2881906275648275 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2898773734517248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚624955944559104 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21796745262429 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152161471 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 338136651 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2898773376053760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2876944033631744 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185985323 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338146796 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44848095 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2881906633248909 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21829683846144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2860109870947120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44848377 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21951519016960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2876944033631744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 _runtime‚17068.006672789 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32794152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77642529 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77642244 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2860109887986480 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152161473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32794149 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21829342422016 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21951630577664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 % _timestamp‚1.7845238944312365e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21829683846144‹ á3‘Õ4Å:/ _wandbruntime‚17068  _runtime‚17068ëí×'Æ:N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚624989106216960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21952629794816 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32795883 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77646259 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21797513089407 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185984836 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2860257301614592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152169264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _step‚1548 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185984836 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338164259 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚646941736011776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2877092325173248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 338154100 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2860257318653952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2882054814703999 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21797853650793 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21830793952256 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 338164259 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2898922760654848 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚646941872468992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚18799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32795886 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2877092308133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44850376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 338154100 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21830452520960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2877092325173248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77646259 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21797513089407 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185984836 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚293 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2860257301614592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152169264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _step‚1548 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185984836 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338164259 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚646941736011776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2877092325173248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 338154100 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2860257318653952 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2882054814703999 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.lªPg;Farge_pool.peak‚ 23175647232 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21797853650793 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚21830793952256 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 338164259 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 136457216 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2898922760654848 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚646941872468992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚18799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32795886 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2877092308133888 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44850376 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 338154100 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚21830452520960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2877092325173248 F&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8672686219215393 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25359195470809937 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6470658183097839 (particle/accuracy‚0.8528761267662048 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26169267296791077 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 *jet/accuracy_2_of_4‚0.7277573943138123 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6724688410758972 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9942034482955933 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5794984698295593 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_3_of_3‚0.9994120597839355 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.710910439491272 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.6523899435997009 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07770160585641861 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2260228991508484 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7845239115199437e+09 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27943044900894165 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6002958416938782 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47899967432022095 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8389833569526672 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7255551218986511 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7107879519462585 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3168264925479889 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05961425602436066 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2733738422393799 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6235854029655457 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2958148121833801 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5691673755645752 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7300859689712524 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5969067215919495 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22854368388652802 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03947204351425171 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9981684684753418 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 /particle/accuracy_4_of_4‚0.9579095244407654 jet/accuracy_2_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07228116691112518 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6243745684623718 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6488826274871826 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9979742765426636 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6242539286613464 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1 jet/accuracy_4_of_5‚0 epoch‚46 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6270816326141357 _runtime‚17085.096010272 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30197757482528687 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9092074036598206 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6822015643119812 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6979955434799194 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2987675666809082 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21718299388885498 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9975854754447937 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28778302669525146 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07181259244680405 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5748507380485535 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4799939692020416 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44789955019950867 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 _step‚1550 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8410053253173828 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21643754839897156 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26266542077064514 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11489113420248032 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2269812822341919 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2846890985965729 +jet/accuracy_3_of_4‚0.08844424784183502 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5823320150375366 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2959422469139099 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05961425602436066 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861111044883728 *jet/accuracy_1_of_2‚0.7721795439720154 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6657513976097107 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29048386216163635 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0945466086268425 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_3‚0.9188315868377686 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6497305631637573 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34790220856666565 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5972413420677185 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28778302669525146 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05915321782231331 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5665980577468872 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6190081834793091 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04883560165762901 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7088865041732788 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7788439989089966 +jet/accuracy_2_of_2‚0.16067849099636078 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9931437373161316 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3441650867462158 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8641345500946045 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 jet/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20979009568691254 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30038654804229736 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 +jet/accuracy_1_of_1‚0.33051052689552307 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.623197615146637 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26954761147499084 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5768284201622009 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.48105159401893616 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5768284201622009 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6892980933189392 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7788439989089966 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8866800665855408 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9488723278045654 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5038282871246338 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6395851969718933 +jet/accuracy_3_of_3‚0.07473097741603851 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21379370987415314 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46699851751327515 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_4_of_5‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07691683620214462 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2381593883037567 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5977612733840942 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3379105031490326 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34354934096336365 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 trainer/global_step‚18799 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27295899391174316 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7162436842918396 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997935950756073 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_1_of_4‚1 *particle/specificity‚0.910302460193634 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6393557786941528 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.050312142819166183 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7530056834220886 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29861655831336975 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6499325037002563 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 *jet/accuracy_1_of_4‚0.9672623872756958 +particle/sensitivity‚0.6429044604301453 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7221358418464661“(j ‚Ñ:¯z & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6499325037002563 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 *jet/accuracy_1_of_4‚0.9672623872756958 +particle/sensitivity‚0.6429044604301453 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7221358418464661 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚7.416382677192028 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6689298748970032 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7validation_average_jet_accuracy‚0.37161654233932495 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5785581469535828 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5770832896232605 *jet/accuracy_2_of_3‚0.5571630001068115 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28858885169029236 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.657538115978241 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8672686219215393 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25359195470809937 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6470658183097839 (particle/accuracy‚0.8528761267662048 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26169267296791077 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 *jet/accuracy_2_of_4‚0.7277573943138123 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6724688410758972 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9942034482955933 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5794984698295593 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_3_of_3‚0.9994120597839355 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.710910439491272 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.6523899435997009 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07770160585641861 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2260228991508484 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 % _timestamp‚1.7845239115199437e+09 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.27943044900894165 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6002958416938782 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47899967432022095 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8389833569526672 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7255551218986511 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7107879519462585 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_4‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3168264925479889 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05961425602436066 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2733738422393799 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6235854029655457 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2958148121833801 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5691673755645752 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7300859689712524 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5969067215919495 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22854368388652802 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03947204351425171 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9981684684753418 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 /particle/accuracy_4_of_4‚0.9579095244407654 jet/accuracy_2_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07228116691112518 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6243745684623718 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6488826274871826 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9979742765426636 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6242539286613464 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1 jet/accuracy_4_of_5‚0 epoch‚46 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6270816326141357 _runtime‚17085.096010272 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/2FRt0SRqqt0F<Þi·³"Bt/event_proportion‚0.01990327425301075 jet/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30197757482528687 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9092074036598206 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6822015643119812 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6979955434799194 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2987675666809082 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21718299388885498 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9975854754447937 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28778302669525146 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07181259244680405 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5748507380485535 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4799939692020416 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44789955019950867 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 _step‚1550 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8410053253173828 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21643754839897156 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26266542077064514 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11489113420248032 jet/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2269812822341919 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2846890985965729 +jet/accuracy_3_of_4‚0.08844424784183502 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5823320150375366 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2959422469139099 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05961425602436066 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861111044883728 *jet/accuracy_1_of_2‚0.7721795439720154 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6657513976097107 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29048386216163635 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0945466086268425 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_3‚0.9188315868377686 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6497305631637573 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34790220856666565 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5972413420677185 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28778302669525146 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05915321782231331 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5665980577468872 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6190081834793091 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04883560165762901 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7088865041732788 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7788439989089966 +jet/accuracy_2_of_2‚0.16067849099636078 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9931437373161316 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3441650867462158 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8641345500946045 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 jet/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20979009568691254 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30038654804229736 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 +jet/accuracy_1_of_1‚0.33051052689552307 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.623197615146637 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26954761147499084 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5768284201622009 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.48105159401893616 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5768284201622009 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6892980933189392 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7788439989089966 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8866800665855408 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9488723278045654 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5038282871246338 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6395851969718933 +jet/accuracy_3_of_3‚0.07473097741603851 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21379370987415314 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46699851751327515 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_4_of_5‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07691683620214462 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2381593883037567 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5977612733840942 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3379105031490326 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34354934096336365 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 trainer/global_step‚18799 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27295899391174316 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7162436842918396 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997935950756073 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_1_of_4‚1 *particle/specificity‚0.910302460193634 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6393557786941528 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.050312142819166183 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7530056834220886 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29861655831336975 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783Ž ±½=µ…Ò:‚(:ú –ÙöÒÏ…Ì network.recv‚ 4.466268e+06 disk.dm-1.in‚0 disk.nvme0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 proc.cpu.threads‚17% disk./.usageGB‚1168.4345970153809 disk.md0.in‚0 disk.md0.out‚ 739.77734375" disk.nvme0c0n1.in‚ 309.37109375* proc.memory.percent‚18.468022346496582 cpu‚25.699690385962633 network.sent‚ 3.0397669e+07 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚53792.6259765625" disk.dm-0.out‚141830.076171875# disk.dm-1.out‚16138.24462890625) disk.nvme0c0n1.out‚105725.10595703125' disk.nvme0n1.out‚105725.10595703125# memory_percent‚97.0782995223999) disk./.usagePercent‚76.10712242405998 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme1c1n1.out‚53792.6259765625( proc.memory.availableMB‚ 478.69140625# proc.memory.rssMB‚ 3025.80078125>,ã‹“Ó:‚(:ˆ –ÙöÒœÉÒÌ gpu.0.gpu‚94.0 gpu.0.memory‚35, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.312' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YK1Ô:‚(:ô ¥ÙöÒêÓ¸Ë" disk.nvme0c0n1.in‚ 309.37109375 proc.cpu.threads‚17 network.sent‚ 3.0442477e+07$ disk./.usageGB‚1168.434585571289 disk.md0.out‚ 740.25390625& disk.nvme0c0n1.out‚105727.23046875$ disk.nvme0n1.out‚105727.23046875# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 cpu‚25.699473645954576 network.recv‚ 4.471179e+06 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚141831.96337890625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375& disk.nvme1c1n1.out‚53793.537109375( proc.memory.availableMB‚ 478.31640625) disk./.usagePercent‚76.10712167863795 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚53793.537109375 disk.dm-1.in‚0# disk.dm-1.out‚16138.36376953125 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.08058834075928:óÛžÕ:‚(:“ ¥ÙöÒÞŠ€Ì gpu.0.gpu‚58.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.409' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.46966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶d;4Ö:/ _wandbruntime‚17127  _runtime‚17127ø¸6}È×: & lr-AdamW/pg2‚5.213873823882607e-06 trainer/global_step‚18849 # _timestamp‚1.78452395381861e+09 _runtime‚17127.395046098 _step‚1551 & lr-AdamW/pg1‚5.213873823882607e-06p÷;eÍØ:Ç # _timestamp‚1.78452395381861e+09 _runtime‚17127.395046098 _step‚1551 & lr-AdamW/pg1‚5.213873823882607e-06 & lr-AdamW/pg2‚5.213873823882607e-06 trainer/global_step‚18849 Q7èÇ4Ù:/ _wandbruntime‚17127  _runtime‚17127¾Àd¤ (Ú:†P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2886300102614528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚648665580944384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2869402660328944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2908205331974144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339215255 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21905570783744 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21905229359616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚648665706923008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _step‚1552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186541326 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2908205690437632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32884965 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186541326 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44996639 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44996367 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2886300102614528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21905570783744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22028062831104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2891274852092404 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2891275209693038 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2908205690437632 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152673931 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186551469 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77881607 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ñ³Ï·(Û:‹P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186541326 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44996639 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44996367 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2886300102614528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21905570783744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22028062831104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2891274852092404 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2891275209693038 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2908205690437632 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152673931 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186551469 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77881607 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2886300102614528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚648665580944384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2869402660328944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2908205331974144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339215255 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21905570783744 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21905229359616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚648665706923008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _step‚1552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186541326 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2908205690437632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32884965 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ³”eO4Ü:/ _wandbruntime‚17128  _runtime‚17128®[‚þoÝ:N ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22029172167168 RDDevicŸ(,.§&eStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _step‚1553 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339242863 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186561139 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44998591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7845239544344869e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152681724 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626670716995584 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21873300151712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186550982 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2869550108035776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152681722 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚18849 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77885290 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2908354716575232 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2886448394156032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2891423391148128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152681722 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22029285817856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21906680889856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44998886 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2869550090996416 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2886448377116672 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21873640713098 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2891423748748874 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚17128.010282256 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21906339458560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77885290 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2908354716575232 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2886448394156032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2891423391148128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152681722 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22029285817856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21906680889856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44998886 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2869550090996416 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2886448377116672 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21873640713098 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2891423748748874 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚17128.010282256 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21906339458560 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339242863 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186561139 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44998591 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7845239544344869e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152681724 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626670716995584 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21873300151712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 186550982 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2869550108035776 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152681722‘ G†WÊ4ß:/ _wandbruntime‚17128  _runtime‚17128{Šï à:… epoch‚47 0loss/FRt1/assignment_loss‚1.4590282266293098 2loss/FBt1/assignment_loss‚1.86638497410619e-05 /loss/FRt2/detection_loss‚1.4987311776222025 _runtime‚17128.011376108 _step‚1554 2loss/SRqqt1/assignment_loss‚0.3881095290046892 3loss/FBt2/assignment_loss‚4.979508692573828e-06 $ _timestamp‚1.784523954435994e+09 %loss/total_loss‚6.992814429250643 0loss/FRt2/assignment_loss‚0.6253788219719398 0loss/SRqqt2/assignment_loss‚0.51223368874642 1loss/SRqqt1/detection_loss‚0.6357972405599759 1loss/SRqqt2/detection_loss‚0.6562406581393613 /loss/FBt1/detection_loss‚0.4253530949541979 /loss/FRt1/detection_loss‚0.4589841800054207 /loss/FBt2/detection_loss‚0.3329341682586925 trainer/global_step‚18849sTŽJá:Š /loss/FRt1/detection_loss‚0.4589841800054207 /loss/FBt2/detection_loss‚0.3329341682586925 trainer/global_step‚18849 epoch‚47 0loss/FRt1/assignment_loss‚1.4590282266293098 2loss/FBt1/assignment_loss‚1.86638497410619e-05 /loss/FRt2/detection_loss‚1.4987311776222025 _runtime‚17128.011376108 _step‚1554 2loss/SRqqt1/assignment_loss‚0.3881095290046892 3loss/FBt2/assignment_loss‚4.979508692573828e-06 $ _timestamp‚1.784523954435994e+09 %loss/total_loss‚6.992814429250643 0loss/FRt2/assignment_loss‚0.6253788219719398 0loss/SRqqt2/assignment_loss‚0.51223368874642 1loss/SRqqt1/detection_loss‚0.6357972405599759 1loss/SRqqt2/detection_loss‚0.6562406581393613 /loss/FBt1/detection_loss‚0.4253530949541979’ ÓÉzâ:‚(:ï ´ÙöÒ諆Ì$ memory_percent‚97.08092212677002 network.sent‚ 3.0458343e+07 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875# proc.memory.rssMB‚ 3025.80078125 cpu‚25.699209757688305) disk./.usagePercent‚76.10712366643004& disk.nvme0n1.out‚105728.2607421875" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0$ disk.dm-0.out‚141832.80810546875 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375% disk./.usageGB‚1168.4346160888672 disk.md0.out‚ 740.51953125( proc.memory.availableMB‚ 478.26171875* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17 network.recv‚ 4.474902e+06! disk.nvme1c1n1.out‚ 53794.0625 disk.nvme1n1.out‚ 53794.0625# disk.dm-1.out‚16138.48486328125 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚105728.2607421875Š_žã:‚(:“ ´ÙöÒè³ÉÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.591' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.53033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+)¤‚ä:‚(:÷ ÃÙöÒ™ýÍË( disk.nvme0c0n1.out‚105729.3251953125& disk.nvme0n1.out‚105729.3251953125% disk.nvme1n1.out‚53794.8056640625# disk.dm-1.out‚16138.63623046875& proc.memory.availableMB‚ 478.484375# proc.memory.rssMB‚ 3025.80078125 cpu‚25.698964174528207 network.sent‚ 3.050895e+07 network.recv‚ 4.478962e+06% disk./.usageGB‚1168.4346961975098 disk.nvme0n1.in‚ 309.37109375 disk.md0.out‚ 741.01171875 disk.rbd0.in‚0 proc.cpu.threads‚17" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚53794.8056640625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚141833.41357421875$ memory_percent‚97.07956314086914( disk./.usagePercent‚76.1071288843843" disk.nvme0c0n1.in‚ 309.37109375 disk.dm-1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚18.468022346496582Î þzžå:‚(:“ ÃÙöÒàõŠÌ gpu.0.gpu‚71.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚287.898' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.96600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þjD2zæ:‚(:ï ÒÙöÒìʾË) disk./.usagePercent‚76.10713211454645 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚741.375 disk.rbd0.out‚0' disk.nvme1c1n1.out‚53795.5712890625$ memory_percent‚97.07958698272705( proc.memory.availableMB‚ 478.48046875 proc.cpu.threads‚17 network.recv‚ 4.482564e+06 disk.md0.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 cpu‚25.69873263069478 network.sent‚ 3.0513939e+07% disk.nvme1n1.out‚53795.5712890625 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚141835.90869140625 disk.dm-1.in‚0# disk.dm-1.out‚16138.70654296875" disk.nvme1c1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3025.80078125% disk./.usageGB‚1168.4347457885742% disk.nvme0c0n1.out‚105731.9140625 disk.nvme0n1.in‚ 309.37109375# disk.nvme0n1.out‚105731.9140625* proc.memory.percent‚18.468022346496582*|ö6žç:‚(:“ ÒÙöÒ¾ÎˆÌ gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚127.319' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚42.43966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±Ö·ò4è:/ _wandbruntime‚17170  _runtime‚17170DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21928015538142 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187025978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22084027174912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78083595 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2876774210741904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚650380447231488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚650380577404416 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚17170.546052973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2915674984646656 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187036121 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚628296550229504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _step‚1556 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45111635 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2898701868679412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2915674626183168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21960798461440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153063579 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45111927 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21961139885568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928|鈌(í:‹P E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78083595 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2876774210741904 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚650380447231488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚650380577404416 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 _runtime‚17170.546052973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2915674984646656 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187036121 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚628296550229504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _step‚1556 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45111635 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2898701868679412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2915674626183168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚21960798461440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153063579 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45111927 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚21961139885568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153063581 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚292 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 340089557 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32971665 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚21961139885568 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22083915614208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2893713827721728 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153063579 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 111560704 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21928015538142 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187025978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22084027174912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184” FË4î:/ _wandbruntime‚17171  _runtime‚17171çŠ(T'ï:N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2876921624370016 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153071374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22085138191360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚18899 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _step‚1557 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32973399 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22085039220736 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21961908560384 _runtime‚17171.156308282 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187035634 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚650414751108096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2898850407735136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153071372 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2915824369254912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚628329711887360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340117165 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340107006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2876921641409376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2893862102223872 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2915824369254912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚628329730499584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340107006 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78087355 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340117165 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45114236 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153071372 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2898850765335882 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2893862119263232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32973402 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187045791 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21929123926506 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45113956 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650414868690944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21928783365120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21962249991680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 % _timestamp‚1.7845239975805547e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2893862102223872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78087638 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153071374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚21961908560384 _runtime‚17171.156308282 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187035634 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚650414751108096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2898850407735136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153071372 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2915824369254912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚628329711887360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340117165 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340107006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2876921641409376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2893862102223872 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2915824369254912 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚628329730499584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340107006 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78087355 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340117165 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45114236 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153071372 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2898850765335882 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2893862119263232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32973402 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187045791 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚21929123926506 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45113956 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650414868690944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21928783365120 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚21962249991680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 % _timestamp‚1.7845239975805547e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2893862102223872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78087638 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153071374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153071374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22085138191360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚18899 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 _step‚1557• â"4ñ:/ _wandbruntime‚17171  _runtime‚17171±s[Oò:‰ 2loss/SRqqt1/assignment_loss‚0.4108957392357447 1loss/SRqqt2/detection_loss‚0.6194287621772981 /loss/FBt1/detection_loss‚0.3675273124518621 trainer/global_step‚18899 3loss/SRqqt2/assignment_loss‚0.41046377227593206 2loss/FBt2/assignment_loss‚0.002701530298284608 /loss/FRt1/detection_loss‚0.4535789669302348 0loss/SRqqt1/detection_loss‚0.612938181560308 0loss/FBt2/detection_loss‚0.23611337014396364 0loss/FRt2/assignment_loss‚0.5387223655624049 $ _timestamp‚1.784523997581528e+09 _runtime‚17171.156826872 _step‚1558 0loss/FRt1/assignment_loss‚1.4838655002874221 3loss/FBt1/assignment_loss‚3.301135005541123e-07 /loss/FRt2/detection_loss‚1.3980818793123555 &loss/total_loss‚6.5343177103493115 epoch‚47~E~çó:Ž 0loss/FRt1/assignment_loss‚1.4838655002874221 3loss/FBt1/assignment_loss‚3.301135005541123e-07 /loss/FRt2/detection_loss‚1.3980818793123555 &loss/total_loss‚6.5343177103493115 epoch‚47 2loss/SRqqt1/assignment_loss‚0.4108957392357447 1loss/SRqqt2/detection_loss‚0.6194287621772981 /loss/FBt1/detection_loss‚0.3675273124518621 trainer/global_step‚18899 3loss/SRqqt2/assignment_loss‚0.41046377227593206 2loss/FBt2/assignment_loss‚0.002701530298284608 /loss/FRt1/detection_loss‚0.4535789669302348 0loss/SRqqt1/detection_loss‚0.612938181560308 0loss/FBt2/detection_loss‚0.23611337014396364 0loss/FRt2/assignment_loss‚0.5387223655624049 $ _timestamp‚1.784523997581528e+09 _runtime‚17171.156826872 _step‚1558– ¼/«†ô:‚(:û áÙöÒµìºË$ disk.dm-0.out‚141836.51025390625* proc.memory.percent‚18.468022346496582 cpu‚25.698501096860777 proc.cpu.threads‚17 network.sent‚ 3.0516861e+07 network.recv‚ 4.485966e+06 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme1c1n1.out‚53796.42724609375 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0$ memory_percent‚97.08013534545898" disk.nvme0c0n1.in‚ 309.37109375) disk.nvme0c0n1.out‚105732.97705078125' disk.nvme0n1.out‚105732.97705078125" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚53796.42724609375 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚16138.94482421875# proc.memory.rssMB‚ 3025.80078125% disk./.usageGB‚1168.4340858459473& proc.memory.availableMB‚ 478.390625( disk./.usagePercent‚76.1070891285424 disk.nvme0n1.in‚ 309.37109375 disk.md0.in‚0 disk.md0.out‚ 741.8828125Û/p“õ:‚(:ˆ áÙöÒŠ×³Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚288.837' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚96.279 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù|/€ö:‚(:õ ðÙöÒ•ÝèË proc.cpu.threads‚17$ disk./.usageGB‚1168.434196472168 disk.rbd0.in‚0) disk.nvme0c0n1.out‚105734.79248046875 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0' disk.nvme0n1.out‚105734.79248046875" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚53796.9599609375 disk.dm-1.in‚0$ memory_percent‚97.08030223846436# proc.memory.rssMB‚ 3025.80078125 cpu‚25.698239753629885 network.sent‚ 3.0581236e+07 network.recv‚ 4.491417e+06) disk./.usagePercent‚76.10709633428876 disk.rbd0.out‚0% disk.nvme1n1.out‚53796.9599609375 disk.dm-0.in‚ 539.2421875! disk.dm-0.out‚141838.34765625# disk.dm-1.out‚16139.04638671875" disk.nvme0c0n1.in‚ 309.37109375 disk.md0.out‚742.0625( proc.memory.availableMB‚ 478.36328125* proc.memory.percent‚18.468022346496582•Dn,“÷:‚(:ˆ ðÙöÒ¸¤§Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚283.914' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.638 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R Ó…€ø:‚(:õ ÿÙöÒ·ýÖË disk.dm-1.out‚16139.12109375) disk.nvme0c0n1.out‚105735.73681640625* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17 network.sent‚ 3.0586179e+07 network.recv‚ 4.494359e+06) disk./.usagePercent‚76.10712192711195% disk./.usageGB‚1168.4345893859863 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚53797.64697265625 disk.md0.out‚ 742.328125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚53797.64697265625 disk.md0.in‚0( proc.memory.availableMB‚ 478.36328125# proc.memory.rssMB‚ 3025.80078125 cpu‚25.698023957082615 disk.dm-0.in‚ 539.2421875 disk.dm-0.out‚ 141839.328125 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0n1.out‚105735.73681640625$ memory_percent‚97.08030223846436£oDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45226435 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2923144278855680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153453231 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21983158190276 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187510630 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 340963859 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153453229 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2901127552828928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22016708987392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22139895379456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2906128885266420 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2884145727076144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚652095459112960 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚652095337328640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187510630 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153453229 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2901127569868288 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21983498751550 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2901127552828928 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22139792207360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚1560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33058368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2923143920392192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2901127569868288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22016367563264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78285093 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33058365}êŸ(ÿ:ŠP KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187520773 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2923143920392192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2901127569868288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22016367563264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78285093 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33058365 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚629955563733504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2923144278855680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22016708987392 % _timestamp‚1.7845240402537932e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚629955545121280 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚17213.83304219 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78284800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2923143920392192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153453231 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45226435 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2923144278855680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153453231 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚21983158190276 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187510630 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 340963859 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153453229 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2901127552828928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22016708987392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22139895379456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2906128885266420 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2884145727076144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚652095459112960 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚652095337328640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187510630 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153453229 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2901127569868288 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚21983498751550 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2901127552828928 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22139792207360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚1560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33058368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33060102 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22140903721984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340991467 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187530443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33060099 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45228771 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22141013178368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22017477662208 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2923293304993280 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78289161 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153461022 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22017819093504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78288870 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340981308 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚1561 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153461022 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2884293157743616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2901275827331072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340991467 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2906277424322144 % _timestamp‚1.7845240408722003e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340981308 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153461024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚18949 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187530443 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2901275827331072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22017819093504 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2906277781922890 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153461024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚652129629113344 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21984266578528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2906277781922890 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153461024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚652129629113344 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚21984266578528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33060102 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22140903721984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340991467 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187530443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33060099 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45228771 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22141013178368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22017477662208 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2923293304993280 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78289161 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153461022 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22017819093504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78288870 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340981308 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚1561 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153461022 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2884293157743616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2901275827331072 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340991467 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚291 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2906277424322144 % _timestamp‚1.7845240408722003e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340981308 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153461024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 trainer/global_step‚18949 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187530443 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534™ ³àR-4ƒ;/ _wandbruntime‚17214  _runtime‚17214‰¿#É„;Š 0loss/FRt2/assignment_loss‚0.7429185715459642 /loss/FRt2/detection_loss‚1.4943909493563252 %loss/total_loss‚6.996472969863582 trainer/global_step‚18949 % _timestamp‚1.7845240408732183e+09 _runtime‚17214.448586987 0loss/FRt1/assignment_loss‚1.5046384825625203 3loss/FBt1/assignment_loss‚0.0002529276228556271 3loss/FBt2/assignment_loss‚1.663874869835395e-05 0loss/SRqqt1/detection_loss‚0.569804591153198 epoch‚47 3loss/SRqqt1/assignment_loss‚0.37565416475149505 3loss/SRqqt2/assignment_loss‚0.43378469782193896 .loss/FRt1/detection_loss‚0.444933243370835 1loss/SRqqt2/detection_loss‚0.6436287186131574 0loss/FBt2/detection_loss‚0.42280454171652143 _step‚1562 /loss/FBt1/detection_loss‚0.3636454426000722oÈ…; 0loss/FRt2/assignment_loss‚0.7429185715459642 /loss/FRt2/detection_loss‚1.4943909493563252 %loss/total_loss‚6.996472969863582 trainer/global_step‚18949 % _timestamp‚1.7845240408732183e+09 _runtime‚17214.448586987 0loss/FRt1/assignment_loss‚1.5046384825625203 3loss/FBt1/assignment_loss‚0.0002529276228556271 3loss/FBt2/assignment_loss‚1.663874869835395e-05 0loss/SRqqt1/detection_loss‚0.569804591153198 epoch‚47 3loss/SRqqt1/assignment_loss‚0.37565416475149505 3loss/SRqqt2/assignment_loss‚0.43378469782193896 .loss/FRt1/detection_loss‚0.444933243370835 1loss/SRqqt2/detection_loss‚0.6436287186131574 0loss/FBt2/detection_loss‚0.42280454171652143 _step‚1562 /loss/FBt1/detection_loss‚0.3636454426000722š .€ô‘…†;‚(:ú ŽÚöÒ¾¨´Ë( disk.nvme1c1n1.out‚53798.47412109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0$ memory_percent‚97.07140922546387' proc.memory.availableMB‚ 479.8203125 proc.cpu.threads‚17 network.sent‚ 3.0589101e+07% disk./.usageGB‚1168.4345512390137 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375* proc.memory.percent‚18.468022346496582 cpu‚25.697750955290612 network.recv‚ 4.498058e+06 disk.dm-1.in‚0 disk.dm-1.out‚16139.22265625 disk.rbd0.out‚0' disk.nvme0n1.out‚105738.33837890625) disk./.usagePercent‚76.10711944237184) disk.nvme0c0n1.out‚105738.33837890625 disk.nvme0n1.in‚ 309.37109375& disk.nvme1n1.out‚53798.47412109375 disk.md0.out‚ 742.83203125# proc.memory.rssMB‚ 3025.80078125# disk.dm-0.out‚141841.5869140625" disk.nvme1c1n1.in‚ 229.87109375%=œU‘‡;‚(:† ŽÚöÒàê¶Ë gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚209.493' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚69.831 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³^$X{ˆ;‚(:ð ÚöÒ¾ó¦Ë# proc.memory.rssMB‚ 3025.80078125 disk.dm-1.in‚0' disk.nvme0c0n1.out‚105739.541015625 disk.nvme0n1.in‚ 309.37109375' disk.nvme1c1n1.out‚53799.2509765625% disk.nvme1n1.out‚53799.2509765625 disk.md0.in‚0 disk.md0.out‚ 743.29296875' proc.memory.availableMB‚ 480.0703125% disk./.usageGB‚1168.4338417053223 disk.dm-0.in‚ 539.2421875 disk.dm-1.out‚ 16139.40625$ memory_percent‚97.06988334655762 proc.cpu.threads‚17 network.sent‚ 3.0651556e+07 network.recv‚ 4.503015e+06) disk./.usagePercent‚76.10707322620566# disk.dm-0.out‚141842.3916015625 disk.rbd0.in‚0% disk.nvme0n1.out‚105739.541015625* proc.memory.percent‚18.468022346496582 cpu‚25.69750686734576 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375×ìò…ž‰;‚(:“ ÚöÒŒ–ÿË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚279.377' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.12566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!ŽÜ…Š;‚(:ú ¬ÚöÒÑÉÔË network.recv‚ 4.506807e+06 disk.nvme0n1.in‚ 309.37109375% disk.nvme1n1.out‚53810.7001953125$ disk.dm-0.out‚141849.16162109375# disk.dm-1.out‚16146.47802734375 disk.rbd0.out‚0' disk.nvme1c1n1.out‚53810.7001953125$ memory_percent‚97.06988334655762# proc.memory.rssMB‚ 3025.80078125% disk./.usageGB‚1168.4339218139648" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚105747.7900390625& disk.nvme0n1.out‚105747.7900390625 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0* proc.memory.percent‚18.468022346496582) disk./.usagePercent‚76.10707844415991" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 746.19140625 disk.rbd0.in‚0' proc.memory.availableMB‚ 480.0703125 cpu‚25.697261774073823 proc.cpu.threads‚17 network.sent‚ 3.0657542e+07)´?rž‹;‚(:“ ¬ÚöÒËü®Ì gpu.0.gpu‚47.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.605' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.20166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…<ã4Œ;/ _wandbruntime‚17256  _runtime‚17256ê 5Ç;Á trainer/global_step‚18999 $ _timestamp‚1.784524083205348e+09 _runtime‚17256.780942765 _step‚1563 % lr-AdamW/pg1‚3.94619293346969e-06 % lr-AdamW/pg2‚3.94619293346969e-06½øýÂÌŽ;Æ _runtime‚17256.780942765 _step‚1563 % lr-AdamW/pg1‚3.94619293346969e-06 % lr-AdamW/pg2‚3.94619293346969e-06 trainer/global_step‚18999 $ _timestamp‚1.784524083205348e+09› õ—f4;/ _wandbruntime‚17256  _runtime‚17256B  (;†P KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2908541277936128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22072278089216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45340714 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2930613214601216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188005425 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78485484 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2908541294975488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153842881 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45340419 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2891517260449744 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78485782 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153842881 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22071936665088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188005425 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2908541277936128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22038641403684 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22195731561984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22072278089216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33145065 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2930613214601216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33145068 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚631614577237504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153842879 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187995282 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚653810180723712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153842879 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _runtime‚17256.781787286 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187995282 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22071936665088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 341848306 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2891517277489104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _step‚1564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341838161 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2908541294975488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22195622098432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2930613573064704 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2913555901853428 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22038981964958 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0à‘3Í‘;‹P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22038641403684 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22195731561984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22072278089216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33145065 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2930613214601216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33145068 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚631614577237504 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153842879 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187995282 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚653810180723712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153842879 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 _runtime‚17256.781787286 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187995282 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22071936665088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 341848306 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2891517277489104 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _step‚1564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341838161 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2908541294975488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22195622098432 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2930613573064704 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2913555901853428 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22038981964958 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2908541277936128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22072278089216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45340714 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2930613214601216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188005425 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78485484 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2908541294975488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153842881 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45340419 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2891517260449744 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78485782 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153842881 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22071936665088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188005425 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 128075776 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2908541277936128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896œ †—B(4’;/ _wandbruntime‚17257  _runtime‚17257ýÉI'“;N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188004938 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188015095 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33146802 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22073388195328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _runtime‚17257.396844476 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22073046764032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45342947 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341865769 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚631647757507584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚631647738895360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22196736407040 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚1565 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22196847960576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341855610 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22073388195328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚653844475302400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2891664691117216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2930762599202304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153850672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78489749 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22040090353322 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚18999 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2908689552438272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78489452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188015095 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188004938 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153850674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845240838211205e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153850674 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22039749791936 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2913704440909152ŠMû'”;•N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2908689552438272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78489452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188015095 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188004938 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153850674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845240838211205e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153850674 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22039749791936 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2913704440909152 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188004938 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188015095 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33146802 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 111553536 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22073388195328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 _runtime‚17257.396844476 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22073046764032 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45342947 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341865769 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚631647757507584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚631647738895360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22196736407040 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚1565 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22196847960576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341855610 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22073388195328 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚653844475302400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2891664691117216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2930762599202304 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153850672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78489749 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22040090353322 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚18999 ¼Gi34•;/ _wandbruntime‚17257  _runtime‚17257aØÁ± –;… epoch‚47 /loss/FRt1/assignment_loss‚1.435596854649773 0loss/FRt2/assignment_loss‚0.6807870587771511 3loss/FBt1/assignment_loss‚0.0018384603736866438 -loss/FRt2/detection_loss‚1.49747675522986 /loss/SRqqt1/detection_loss‚0.63968507016073 /loss/FBt1/detection_loss‚0.3199876395283209 _step‚1566 2loss/SRqqt1/assignment_loss‚0.4069530183583843 1loss/SRqqt2/assignment_loss‚0.419195939901159 4loss/FBt2/assignment_loss‚1.7297644185557708e-05 0loss/FRt1/detection_loss‚0.45185058818046925 0loss/SRqqt2/detection_loss‚0.649723698264733 0loss/FBt2/detection_loss‚0.33262340075249763 %loss/total_loss‚6.835735781820951 trainer/global_step‚18999 % _timestamp‚1.7845240838221002e+09 _runtime‚17257.397338257ó(vÚ—;Š 2loss/SRqqt1/assignment_loss‚0.4069530183583843 1loss/SRqqt2/assignment_loss‚0.419195939901159 4loss/FBt2/assignment_loss‚1.7297644185557708e-05 0loss/FRt1/detection_loss‚0.45185058818046925 0loss/SRqqt2/detection_loss‚0.649723698264733 0loss/FBt2/detection_loss‚0.33262340075249763 %loss/total_loss‚6.835735781820951 trainer/global_step‚18999 % _timestamp‚1.7845240838221002e+09 _runtime‚17257.397338257 epoch‚47 /loss/FRt1/assignment_loss‚1.435596854649773 0loss/FRt2/assignment_loss‚0.6807870587771511 3loss/FBt1/assignment_loss‚0.0018384603736866438 -loss/FRt2/detection_loss‚1.49747675522986 /loss/SRqqt1/detection_loss‚0.63968507016073 /loss/FBt1/detection_loss‚0.3199876395283209 _step‚1566ž •ÛüƇ˜;‚(:ü »ÚöÒÚÔÍË disk.dm-0.in‚ 539.2421875& disk.nvme1n1.out‚53811.81201171875 disk.md0.out‚ 746.49609375 disk.rbd0.out‚0' disk.nvme0n1.out‚105748.84912109375$ memory_percent‚97.07009792327881* proc.memory.percent‚18.468022346496582 cpu‚25.6970473214087$ disk.dm-0.out‚141850.32177734375) disk./.usagePercent‚76.10711720610574( disk.nvme1c1n1.out‚53811.81201171875 disk.md0.in‚0 disk.rbd0.in‚0) disk.nvme0c0n1.out‚105748.84912109375( proc.memory.availableMB‚ 480.03515625 proc.cpu.threads‚17 network.recv‚ 4.51021e+06 disk.dm-1.in‚0# disk.dm-1.out‚16146.83154296875" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375# proc.memory.rssMB‚ 3025.80078125 network.sent‚ 3.0662493e+07% disk./.usageGB‚1168.4345169067383" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375UkÏz“™;‚(:ˆ »ÚöÒ­÷ÌÌ gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚261.975' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚87.325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦V¨š;‚(:ó ÊÚöÒ‡±ÓË" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3025.80078125 proc.cpu.threads‚17 network.sent‚ 3.0725742e+07 network.recv‚ 4.515245e+06% disk./.usageGB‚1168.4346466064453$ disk.dm-0.out‚141854.44775390625 disk.md0.out‚ 746.82421875# memory_percent‚97.0700740814209 cpu‚25.696789571498787 disk.dm-0.in‚ 539.2421875# disk.dm-1.out‚16146.86279296875% disk.nvme0c0n1.out‚105753.0078125 disk.nvme0n1.in‚ 309.37109375' proc.memory.availableMB‚ 480.0390625* proc.memory.percent‚18.468022346496582) disk./.usagePercent‚76.10712565422213 disk.rbd0.out–ÿ$ëÖ‚0# disk.nvme0n1.out‚105753.0078125' disk.nvme1c1n1.out‚53812.5244140625 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚53812.5244140625 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0³ƒÝ~“›;‚(:ˆ ÊÚöÒçʬÌ gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚280.827' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.609 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕF }œ;‚(:ò ÙÚöÒúηË network.recv‚ 4.518254e+06) disk./.usagePercent‚76.10712441185208$ disk./.usageGB‚1168.434627532959 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚ 229.87109375 network.sent‚ 3.0728706e+07 disk.md0.out‚ 747.265625$ disk.dm-0.out‚141855.54736328125 disk.dm-1.out‚16147.17578125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0c0n1.out‚105754.51953125$ disk.nvme0n1.out‚105754.51953125 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚53813.37451171875& disk.nvme1n1.out‚53813.37451171875' proc.memory.availableMB‚ 480.0078125* proc.memory.percent‚18.468022346496582 cpu‚25.696576402506782" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.07026481628418# proc.memory.rssMB‚ 3025.80078125 proc.cpu.threads‚17~T?ž;‚(:“ ÙÚöÒþ«‰Ì gpu.0.gpu‚91.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚271.512' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.50399999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0. 9,4ž;/ _wandbruntime‚17299  _runtime‚17299ÌDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78685977 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2920982918440436 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22127847191040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2898888810862704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚633273590741504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2938082508810240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33231765 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2915955003043328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22094124617092 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚1568 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2920983276041070 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188490077 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45454212 % _timestamp‚1.7845241262892478e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154232531 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2915955003043328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2938082508810240 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22127505766912 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22127847191040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22127505766912 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2898888793823344 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154232529 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188479934 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154232531 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2915955003043328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2938082508810240 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22127505766912 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22127847191040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22127505766912 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2898888793823344 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154232529 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188479934 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78685977 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2920982918440436 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22127847191040 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2898888810862704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚633273590741504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2938082508810240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33231765 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2915955003043328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22094124617092 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚1568 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2920983276041070 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452  ¿&‡œ4¤;/ _wandbruntime‚17300  _runtime‚17300ÖàÈØ%¥;N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _runtime‚17300.481541422 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154240324 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2938231893411328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚655559431970816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22252561988608 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45456738 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚655559314387968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚19049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342729912 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2916103294584832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78689961 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22128957297152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2899036224490816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188499747 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188489590 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342740071 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845241269057314e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2921131815096906 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188489590 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2938232251881984 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154240324 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22095233005344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2916103277545472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 _step‚1569 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚2 C‰E=2095573566730 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496íK&Ü'¦;”N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22128957297152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2899036224490816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188499747 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188489590 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342740071 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 % _timestamp‚1.7845241269057314e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2921131815096906 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188489590 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2938232251881984 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154240324 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22095233005344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2916103277545472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 _step‚1569 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22095573566730 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _runtime‚17300.481541422 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154240324 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2938231893411328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚655559431970816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22252561988608 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45456738 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚655559314387968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚19049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342729912 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2916103294584832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78689961¡ K4µ4§;/ _wandbruntime‚17300  _runtime‚17300§ Rø¨;Œ epoch‚47 3loss/SRqqt2/assignment_loss‚0.42290383699047507 4loss/FBt1/assignment_loss‚2.6499048992492624e-05 _runtime‚17300.482069543 0loss/FRt1/detection_loss‚0.45107973303619275 /loss/FRt2/detection_loss‚1.4663807734650705 1loss/SRqqt2/detection_loss‚0.5629920126707592 /loss/FBt2/detection_loss‚0.3921629093244216 $loss/total_loss‚6.68929460196402 % _timestamp‚1.7845241269068131e+09 /loss/FRt1/assignment_loss‚1.436460003426773 3loss/SRqqt1/assignment_loss‚0.41809577067419657 3loss/FBt2/assignment_loss‚4.070402440585023e-06 1loss/SRqqt1/detection_loss‚0.5440061893771022 trainer/global_step‚19049 _step‚1570 0loss/FRt2/assignment_loss‚0.5857929817165237 0loss/FBt1/detection_loss‚0.40938982183107314Á_Õ ©;‘ 4loss/FBt1/assignment_loss‚2.6499048992492624e-05 _runtime‚17300.482069543 0loss/FRt1/detection_loss‚0.45107973303619275 /loss/FRt2/detection_loss‚1.4663807734650705 1loss/SRqqt2/detection_loss‚0.5629920126707592 /loss/FBt2/detection_loss‚0.3921629093244216 $loss/total_loss‚6.68929460196402 % _timestamp‚1.7845241269068131e+09 /loss/FRt1/assignment_loss‚1.436460003426773 3loss/SRqqt1/assignment_loss‚0.41809577067419657 3loss/FBt2/assignment_loss‚4.070402440585023e-06 1loss/SRqqt1/detection_loss‚0.5440061893771022 trainer/global_step‚19049 _step‚1570 0loss/FRt2/assignment_loss‚0.5857929817165237 0loss/FBt1/detection_loss‚0.40938982183107314 epoch‚47 3loss/SRqqt2/assignment_loss‚0.42290383699047507¢ ÌÚ<ƒª;‚(:ø èÚöÒðÒïË) disk.nvme0c0n1.out‚105758.13134765625 network.sent‚ 3.0746481e+07 disk.md0.in‚0# disk.dm-0.out‚141859.1416015625$ memory_percent‚97.07198143005371 proc.cpu.threads‚17" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875 disk.dm-1.out‚16147.25390625" disk.nvme0c0n1.in‚ 309.37109375' proc.memory.availableMB‚ 479.7265625# proc.memory.rssMB‚ 3025.80078125 cpu‚25.69633275984914) disk./.usagePercent‚76.10717385818042( disk.nvme1c1n1.out‚53814.20458984375& disk.nvme1n1.out‚53814.20458984375 disk.dm-1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚18.468022346496582 network.recv‚ 4.522087e+06% disk./.usageGB‚1168.4353866577148 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚105758.13134765625 disk.md0.out‚ 747.6328125 disk.rbd0.out‚0/µž«;‚(:“ èÚöÒȲÌ gpu.0.gpu‚91.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚272.457' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.81899999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NãNꃬ;‚(:ø ÷ÚöÒºã™Ë disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375' proc.memory.availableMB‚ 479.6953125" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-1.in‚0# disk.dm-1.out‚16147.35107421875* proc.memory.percent‚18.468022346496582 cpu‚25.696121729659207 proc.cpu.threads‚17 network.sent‚ 3.0796342e+07 network.recv‚ 4.527049e+06) disk./.usagePercent‚75.93017737148368 disk.nvme1n1.in‚ 229.87109375) disk.nvme0c0n1.out‚105759.40673828125 disk.nvme0n1.in‚ 309.37109375' disk.nvme0n1.out‚105759.40673828125% disk./.usageGB‚1165.7180480957031# disk.dm-0.out‚141860.0830078125$ memory_percent‚97.07217216491699# proc.memory.rssMB‚ 3025.80078125' disk.nvme1c1n1.out‚53814.8232421875% disk.nvme1n1.out‚53814.8232421875 disk.md0.out‚ 748.03125K1¢÷ž­;‚(:“ ÷ÚöÒ ±õË gpu.0.gpu‚56.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚269.224' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.74133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’U‡•ƒ®;‚(:ø †ÛöÒéÒÅË network.recv‚ 4.530873e+06) disk./.usagePercent‚75.93015972982884% disk.nvme1n1.out‚53815.4111328125 disk.dm-0.in‚ 539.2421875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚105763.1513671875 disk.nvme0n1.in‚ 309.37109375% disk./.usageGB‚1165.7177772521973 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-1.in‚0# disk.dm-1.out‚16147.40185546875 disk.rbd0.in‚0' disk.nvme1c1n1.out‚53815.4111328125$ memory_percent‚97.07393646240234& disk.nvme0n1.out‚105763.1513671875% proc.memory.availableMB‚ 479.40625# proc.memory.rssMB‚ 3025.80078125 network.sent‚ 3.0802305e+07 disk.md0.out‚ 748.29296875$ disk.dm-0.out‚141863.79833984375" disk.nvme1c1n1.in‚ 229.87109375* proc.memory.percent‚18.468022346496582 cpu‚25.695893569517917 proc.cpu.threads‚17º¼ktž¯;‚(:“ †ÛöÒÜÑ¥Ì gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.589' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.19633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×`°Í4°;/ _wandbruntime‚17343  _runtime‚17343nРȱ; & lr-AdamW/pg1‚3.198428588725716e-06 & lr-AdamW/pg2‚3.198428588725716e-06 trainer/global_step‚19099 $ _timestamp‚1.784524169489813e+09 _runtime‚17343.06705448 _step‚1571 +èºÍ²;Ç & lr-AdamW/pg2‚3.198428588725716e-06 trainer/global_step‚19099 $ _timestamp‚1.784524169489813e+09 _runtime‚17343.06705448 _step‚1571 & lr-AdamW/pg1‚3.198428588725716e-06£ #r‚H4³;/ _wandbruntime‚17343  _runtime‚17343Nöû2 (´;…P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2945552161482752 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 343586765 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188974729 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78887244 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚634932585633280 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22149948391774 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45568776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2945551803019264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22307370051584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188974729 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154622181 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2923368728150528 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154622181 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188964586 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2945551803019264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45568488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188964586 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚17343.06714784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22149607830500 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚657239974297088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 343596910 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33318465 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2923368728150528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22183416292864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚634932604245504 trainer/global_step‚19099‡ Õ.jµ;ŠP MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2923368728150528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22183416292864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚634932604245504 trainer/global_step‚19099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2945552161482752 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 343586765 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188974729 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78887244 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚634932585633280 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22149948391774 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45568776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2945551803019264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22307370051584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188974729 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154622181 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2923368728150528 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154622181 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188964586 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2945551803019264 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45568488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188964586 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚17343.06714784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22149607830500 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚657239974297088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 343596910 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33318465 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799¤ ~C¥4¶;/ _wandbruntime‚17343  _runtime‚17343E´bd'·;N EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 343604214 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚657274138137600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2928558474083168 _runtime‚17343.678892302 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45571047 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33320202 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2945701187620352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154629974 trainer/global_step‚19099 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188984399 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2923517019692032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154629972 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22184184967680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚308 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22150716218752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 343614373 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 % _timestamp‚1.7845241701030927e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚311 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78890938 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚634965765903360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2906407757864416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22184184967680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2945701546091008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188974242 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45570739 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 343604214 _step‚1573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2906407774903776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22184526398976þYz0'¸;•N @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22184184967680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2945701546091008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188974242 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45570739 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 343604214 _step‚1573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2906407774903776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22184526398976 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 343604214 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚657274138137600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2928558474083168 _runtime‚17343.678892302 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45571047 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33320202 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2945701187620352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154629974 trainer/global_step‚19099 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 188984399 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2923517019692032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154629972 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22184184967680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚308 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22150716218752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 343614373 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 % _timestamp‚1.7845241701030927e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚311 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78890938 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚634965765903360 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2906407757864416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 117844992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155011829 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2935836951614452 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2953021455691776 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2930782470297088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22205431605182 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2953021097228288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22363105478144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22205091043908 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2930782453257728 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7845242113978794e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79088300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189459381 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 189449238 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45683135 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155011831 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155011829 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2935837309215086 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22238643970560 _runtime‚17384.976944999 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚636591617749504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22363202358784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2913631860570544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155011829 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115492864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96880640 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2935837309215086 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22238643970560 _runtime‚17384.976944999 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚636591617749504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22363202358784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2913631860570544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155011829 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2935836951614452 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2953021455691776 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2930782470297088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22205431605182 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2953021097228288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22363105478144 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22205091043908 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2930782453257728¨ Añ8érÆ;‚(:ç ³ÛöÒ¤ÇË$ disk.nvme1n1.out‚53819.048828125* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17 network.recv‚ 4.542707e+06 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0 disk.md0.out‚ 749.7421875' disk./.usagePercent‚75.914682780584" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme0n1.out‚ 105773.140625 disk.md0.in‚0$ memory_percent‚97.07708358764648# proc.memory.rssMB‚ 3025.80078125 cpu‚25.6952576220636$ disk./.usageGB‚1165.480167388916 disk.dm-1.in‚0" disk.dm-1.out‚16147.7353515625 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 105773.140625 disk.nvme0n1.in‚ 309.37109375& disk.nvme1c1n1.out‚53819.048828125& proc.memory.availableMB‚ 478.890625 network.sent‚ 3.0886322e+07# disk.dm-0.out‚141874.0068359375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375¡Ù‚{“Ç;‚(:ˆ ³ÛöÒäÃ’Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚154.251' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚51.417 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼74È;/ _wandbruntime‚17385  _runtime‚17385vÅ’'É;N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79092677 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22364323246592 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2913779308277376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33406902 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚636624779407360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189469051 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45685775 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155019624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344478516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79092375 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344488675 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚658988997391872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 % _timestamp‚1.7845242120018616e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2935985490670176 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2930930727759872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344478516 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155019622 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 _runtime‚17385.57766443 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45685476 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189458894 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚299 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2930930744799232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22240095500800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚658989121266176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22364217984512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189469051 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 trainer/global_step‚19149 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22239754069504 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2953170481829376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22240095500800 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚658989121266176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22364217984512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189469051 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 trainer/global_step‚19149 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22239754069504 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2953170481829376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚302 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 TFDeviceStVG½atsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189469051 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45685775 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155019624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344478516 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79092375 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344488675 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚658988997391872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 % _timestamp‚1.7845242120018616e+09© à®4Ë;/ _wandbruntime‚17385  _runtime‚17385}oÉ Ì;‡ 3loss/SRqqt1/assignment_loss‚0.48724781199163236 3loss/SRqqt2/assignment_loss‚0.46025280556489834 /loss/FBt1/detection_loss‚0.2946550969477155 %loss/total_loss‚7.185159158314815 2loss/FBt1/assignment_loss‚0.035368937797315064 3loss/FBt2/assignment_loss‚9.095495448534578e-05 /loss/FRt1/detection_loss‚0.5166759106620841 epoch‚47 0loss/FRt1/assignment_loss‚1.4969884400415903 /loss/FRt2/detection_loss‚1.5040997044015458 0loss/FBt2/detection_loss‚0.47269146960051905 % _timestamp‚1.7845242120073783e+09 _step‚1578 0loss/FRt2/assignment_loss‚0.6395833636439123 .loss/SRqqt1/detection_loss‚0.6175286902841 0loss/SRqqt2/detection_loss‚0.659975972425017 trainer/global_step‚19149 _runtime‚17385.586036467ܸKRÍ;Œ 3loss/SRqqt1/assignment_loss‚0.48724781199163236 3loss/SRqqt2/assignment_loss‚0.46025280556489834 /loss/FBt1/detection_loss‚0.2946550969477155 %loss/total_loss‚7.185159158314815 2loss/FBt1/assignment_loss‚0.035368937797315064 3loss/FBt2/assignment_loss‚9.095495448534578e-05 /loss/FRt1/detection_loss‚0.5166759106620841 epoch‚47 0loss/FRt1/assignment_loss‚1.4969884400415903 /loss/FRt2/detection_loss‚1.5040997044015458 0loss/FBt2/detection_loss‚0.47269146960051905 % _timestamp‚1.7845242120073783e+09 _step‚1578 0loss/FRt2/assignment_loss‚0.6395833636439123 .loss/SRqqt1/detection_loss‚0.6175286902841 0loss/SRqqt2/detection_loss‚0.659975972425017 trainer/global_step‚19149 _runtime‚17385.586036467ª §tÒ„žÎ;‚(:“ ÂÛöÒ¢±ŸË gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚271.309' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.43633333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]7Ê)Ï;‚(:ö ÂÛöÒ€Á¡Ì disk.md0.out‚ 750.26171875 disk.dm-1.in‚0' proc.memory.availableMB‚ 479.0390625# proc.memory.rssMB‚ 3025.80078125 network.recv‚ 4.547482e+06) disk./.usagePercent‚75.91468153821394 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375" disk.dm-1.out‚16147.8720703125* proc.memory.percent‚18.468022346496582 cpu‚25.695044204515558 network.sent‚ 3.0938029e+07% disk./.usageGB‚1165.4801483154297 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375$ disk.nvme0n1.out‚105783.77734375 disk.md0.in‚0$ disk.dm-0.out‚141885.62841796875# memory_percent‚97.0761775970459 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0c0n1.out‚105783.77734375( disk.nvme1c1n1.out‚53821.30224609375& disk.nvme1n1.out‚53821.30224609375¥º¸áŸÐ;‚(:” ÑÛöÒ†“ÜË gpu.0.gpu‚98.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚180.077' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚60.025666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m¥v…Ñ;‚(:ú ÑÛöÒŸªÌ) disk./.usagePercent‚76.05607468071288" disk.nvme0c0n1.in‚ 309.37109375" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚53822.07177734375# disk.dm-0.out‚142936.0068359375# proc.memory.rssMB‚ 3025.80078125 cpu‚25.69484746659184 disk.dm-1.in‚0# disk.dm-1.out‚16148.00537109375 disk.rbd0.out‚0& disk.nvme0n1.out‚106835.3466796875( disk.nvme1c1n1.out‚53822.07177734375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 network.sent‚ 3.097823e+07% disk./.usageGB‚1167.6508865356445 disk.md0.out‚ 750.76953125$ memory_percent‚97.07620143890381( proc.memory.availableMB‚ 479.03515625* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17 network.recv‚ 4.551636e+06 disk.rbd0.in‚0( disk.nvme0c0n1.out‚106835.3466796875 disk.nvme0n1.in‚ 309.37109375 disk.dm-0.in‚ 539.2421875%6M¼4Ò;/ _wandbruntime‚17427  _runtime‚17427OŸ@ÊÓ;Ä & lr-AdamW/pg1‚2.528776287948012e-06 & lr-AdamW/pg2‚2.528776287948012e-06 trainer/global_step‚19199 % _timestamp‚1.7845242540541732e+09 _runtime‚17427.629962638 _step‚1579xa=›ÏÔ;É & lr-AdamW/pg1‚2.528776287948012e-06 & lr-AdamW/pg2‚2.528776287948012e-06 trainer/global_step‚19199 % _timestamp‚1.7845242540541732e+09 _runtime‚17427.629962638 _step‚1579« æf§4Õ;/ _wandbruntime‚17427  _runtime‚17427‘³¹x (Ö;…P OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚638250612641280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2921003393944144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22294554496512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155401479 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155401481 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2960490391437312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155401481 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45798629 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22294213072384 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155401479 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79290220 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2921003410983504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2938196195404288 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2938196178364928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2943263968201460 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33491868 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2938196178364928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _step‚1580 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚638250631253504 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2938196195404288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 % _timestamp‚1.7845242540551994e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 189944033 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79290497 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22419034183680 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2960490391437312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660669665437184 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2960490749900800 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22294554496512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22260574257316 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360Szf (×;ŠP A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 189944033 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79290497 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22419034183680 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2960490391437312 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660669665437184 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2960490749900800 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22294554496512 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22260574257316 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚638250612641280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2921003393944144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22294554496512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155401479 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155401481 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2960490391437312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155401481 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45798629 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22294213072384 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155401479 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79290220 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2921003410983504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 125978624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2938196195404288 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2938196178364928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2943263968201460 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33491868 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2938196178364928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 _step‚1580 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚638250631253504 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2938196195404288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 % _timestamp‚1.7845242540551994e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0¬ Æ©È4Ø;/ _wandbruntime‚17428  _runtime‚17428±@ñïíÙ;N ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22295664602624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22420146840576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660703958364160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22420045772800 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚19199 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚638283792911360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189943546 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638283811523584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660703838684160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚17428.264069873 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155409272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2960639776038400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2938344469906432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155409272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33493602 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189943546 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189953703 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155409274 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22295323171328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2960640134509056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22295323171328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79294225 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2938344452867072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22295664602624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2943412507257184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345352818 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33493599 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2960640134509056 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155409274 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22261682645568 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45800626 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345362977 % _timestamp‚1.7845242546872814e+09 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189953703 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157ñÙXö'Ú;•N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189953703 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22295664602624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22420146840576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660703958364160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22420045772800 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚19199 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚638283792911360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189943546 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638283811523584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660703838684160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 _runtime‚17428.264069873 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155409272 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2960639776038400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2938344469906432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155409272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33493602 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189943546 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189953703 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155409274 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22295323171328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2960640134509056 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚297 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22295323171328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79294225 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2938344452867072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22295664602624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2943412507257184 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345352818 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33493599 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2960640134509056 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155409274 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22261682645568 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45800626 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345362977 % _timestamp‚1.7845242546872814e+09 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272­ ÔÔ›M4Û;/ _wandbruntime‚17428  _runtime‚17428ùÏ ìÜ; 3loss/SRqqt1/assignment_loss‚0.44696639620408585 3loss/FBt2/assignment_loss‚8.153128629986541e-06 1loss/SRqqt2/detection_loss‚0.5863199935397649 /loss/FBt2/detection_loss‚0.2641416263740717 /loss/FRt1/detection_loss‚0.4809734225318395 1loss/SRqqt1/detection_loss‚0.6100524884089827 0loss/FBt1/detection_loss‚0.31585530792124594 %loss/total_loss‚6.699491017300307 % _timestamp‚1.7845242546885998e+09 0loss/FRt1/assignment_loss‚1.4272580712641143 /loss/FRt2/detection_loss‚1.4892727298811859 epoch‚47 0loss/FRt2/assignment_loss‚0.6305007298231681 3loss/SRqqt2/assignment_loss‚0.44771033094080803 4loss/FBt1/assignment_loss‚0.00043176728241081766 trainer/global_step‚19199 _runtime‚17428.265145435 _step‚1582ØëŒÝ;’ /loss/FRt1/detection_loss‚0.4809734225318395 1loss/SRqqt1/detection_loss‚0.6100524884089827 0loss/FBt1/detection_loss‚0.31585530792124594 %loss/total_loss‚6.699491017300307 % _timestamp‚1.7845242546885998e+09 0loss/FRt1/assignment_loss‚1.4272580712641143 /loss/FRt2/detection_loss‚1.4892727298811859 epoch‚47 0loss/FRt2/assignment_loss‚0.6305007298231681 3loss/SRqqt2/assignment_loss‚0.44771033094080803 4loss/FBt1/assignment_loss‚0.00043176728241081766 trainer/global_step‚19199 _runtime‚17428.265145435 _step‚1582 3loss/SRqqt1/assignment_loss‚0.44696639620408585 3loss/FBt2/assignment_loss‚8.153128629986541e-06 1loss/SRqqt2/detection_loss‚0.5863199935397649 /loss/FBt2/detection_loss‚0.2641416263740717® &»þ¢{Þ;‚(:ð àÛöÒÜ€ªÌ$ disk./.usageGB‚1168.479694366455" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582) disk./.usagePercent‚76.11005988382706& disk.nvme1c1n1.out‚53881.267578125 disk.nvme1n1.in‚ 229.87109375$ disk.nvme1n1.out‚53881.267578125 disk.md0.out‚ 751.06640625 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚97.07903861999512 disk.dm-0.in‚ 539.2421875$ disk.dm-0.out‚144052.19482421875" disk.nvme0c0n1.in‚ 309.37109375% disk.nvme0c0n1.out‚107892.3203125 disk.nvme0n1.in‚ 309.37109375' proc.memory.availableMB‚ 478.5703125 cpu‚25.69498092014892 proc.cpu.threads‚17 network.recv‚ 4.554993e+06# disk.dm-1.out‚16148.10302734375# disk.nvme0n1.out‚107892.3203125 network.sent‚ 3.0981152e+07`ŽŸß;‚(:” àÛöÒìÄ°Ë gpu.0.gpu‚97.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚121.248' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.416000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îçÙepà;‚(:å ïÛöÒ’ÞÉË cpu‚25.696963313666625 network.sent‚ 3.1043631e+07" disk.nvme0c0n1.in‚ 309.37109375' disk.nvme0c0n1.out‚107893.794921875" disk.nvme1c1n1.in‚ 229.87109375" proc.memory.availableMB‚478.25) disk./.usagePercent‚76.11006808346946% disk./.usageGB‚1168.4798202514648# disk.dm-1.out‚16148.22314453125 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 proc.cpu.threads‚17 network.recv‚ 4.560193e+06 disk.dm-1.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚107893.794921875! disk.nvme1c1n1.out‚ 53881.8125 disk.nvme1n1.out‚ 53881.8125$ disk.dm-0.out‚144053.36962890625# proc.memory.rssMB‚ 3025.80078125 disk.rbd0.out‚0 disk.md0.out‚ 751.390625 disk.dm-0.in‚ 539.2421875$ memory_percent‚97.08099365234375* proc.memory.percent‚18.468022346496582ÓØ»Oá;‚(:’ ïÛöÒŽ‚¡Ì gpu.0.gpu‚29.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚290.874' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.95800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©0¬4â;/ _wandbruntime‚17445  _runtime‚17445äÆcóµ'ã;¤z ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6473047733306885 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7190730571746826 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 /particle/accuracy_4_of_4‚0.9539412260055542 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6155673265457153 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27441471815109253 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22705480456352234 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04762999713420868 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07662560045719147 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6357756853103638 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 'particle/f_score‚0.6525243520736694 6validation_average_jet_accuracy‚0.3714224398136139 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48240113258361816 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5849531292915344 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07976909726858139 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.29501578211784363 *jet/accuracy_1_of_2‚0.7717509269714355 )jet/accuracy_1_of_4‚0.966553807258606 particle/accuracy_1_of_1‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.264253705739975 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_1‚0.3303734064102173 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9975854754447937 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9488723278045654 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7533763647079468 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6227839589118958 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3383229374885559 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_5_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6710079908370972 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.6426489949226379 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8410053253173828 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20979009568691254 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5753681659698486 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28994059562683105 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7101946473121643 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2942741811275482 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6896573305130005 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22822411358356476 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25384944677352905 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2299940288066864 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9981684684753418 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6690983772277832 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6482939720153809 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5770832896232605 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6825462579727173 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21379370987415314 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2894061505794525 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46699851751327515 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9979742765426636 )loss/val_total_loss‚7.421452397240436 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5753681659698486 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27498453855514526 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997935950756073 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07894910126924515 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2839032709598541 *jet/accuracy_2_of_2‚0.1603655219078064 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6219704747200012 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7154925465583801 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5956888198852539 _step‚1583 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2938852310180664 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7056745290756226 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8662042021751404 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.48105159401893616 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05798707529902458 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3017432689666748 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31536251306533813 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09214159101247787 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6353245973587036 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 jet/accuracy_1_of_5‚1 jet/accuracy_4_of_5‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_2_of_3‚0.5556069612503052 +jet/accuracy_3_of_4‚0.08844424784183502 jet/accuracy_2_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4833953380584717 *jet/accuracy_2_of_4‚0.7326289415359497 ,jet/accuracy_4_of_4‚0.013899499550461769 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9102394580841064 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21780703961849213 *particle/specificity‚0.910559892654419 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6957160234451294 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26984456181526184 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5732604265213013 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_4‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5769678950309753 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6432563066482544 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8866800665855408 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05045656859874725 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7845242715982523e+09 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6250592470169067 9"Purity/2FR" i´ut1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6616311073303223 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06958212703466415 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861111044883728 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚19199 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8389833569526672 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7036715149879456 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44651666283607483 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21835419535636902 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03947649523615837 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30039501190185547 *jet/accuracy_1_of_3‚0.9182965159416199 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚17445.174362976 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5819100141525269 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_2_of_2‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5960009694099426 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07715009897947311 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5652926564216614 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7296767830848694 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2631312906742096 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23926465213298798 +jet/accuracy_3_of_3‚0.07522188872098923 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7251509428024292 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6227388381958008 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6465587615966797 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8641345500946045 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3455651104450226 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34705233573913574 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30029359459877014 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28088632225990295 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5984411835670471 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11256016790866852 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.771223247051239 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6216285824775696 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8530234098434448 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5651360750198364 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3023985028266907 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9942034482955933 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.507670521736145 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 /particle/accuracy_3_of_3‚0.9994120597839355 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9931437373161316 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34272196888923645 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5961403846740723 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06958212703466415 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2894061505794525 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 epoch‚47 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6679396629333496 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.771223247051239 particle/accuracy_1_of_5‚1ÝRv—/=ä;©z ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4833953380584717 *jet/accuracy_2_of_4‚0.7326289415359497 ,jet/accuracy_4_of_4‚0.013899499550461769 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9102394580841064 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21780703961849213 *particle/specificity‚0.910559892654419 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6957160234451294 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.26984456181526184 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5732604265213013 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_4‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5769678950309753 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6432563066482544 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8866800665855408 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05045656859874725 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7845242715982523e+09 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6250592470169067 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6616311073303223 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06958212703466415 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861111044883728 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚19199 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8389833569526672 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7036715149879456 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44651666283607483 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21835419535636902 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03947649523615837 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30039501190185547 *jet/accuracy_1_of_3‚0.9182965159416199 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 _runtime‚17445.174362976 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5819100141525269 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_2_of_2‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5960009694099426 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07715009897947311 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5652926564216614 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7296767830848694 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2631312906742096 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23926465213298798 +jet/accuracy_3_of_3‚0.07522188872098923 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7251509428024292 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6227388381958008 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6465587615966797 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8641345500946045 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3455651104450226 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34705233573913574 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30029359459877014 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28088632225990295 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5984411835670471 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11256016790866852 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.771223247051239 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6216285824775696 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8530234098434448 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5651360750198364 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3023985028266907 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9942034482955933 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.507670521736145 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 /particle/accuracy_3_of_3‚0.9994120597839355 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9931437373161316 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34272196888923645 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5961403846740723 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06958212703466415 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2894061505794525 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 epoch‚47 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6679396629333496 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.771223247051239 particle/accuracy_1_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6473047733306885 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7190730571746826 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 /particle/accuracy_4_of_4‚0.9539412260055542 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6155673265457153 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27441471815109253 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22705480456352234 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04762999713420868 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07662560045719147 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6357756853103638 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_4‚1 'particle/f_score‚0.6525243520736694 6validation_average_jet_accuracy‚0.3714224398136139 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48240113258361816 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5849531292915344 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07976909726858139 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.29501578211784363 *jet/accuracy_1_of_2‚0.7717509269714355 )jet/accuracy_1_of_4‚0.966553807258606 particle/accuracy_1_of_1‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.264253705739975 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_1_of_1‚0.3303734064102173 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9975854754447937 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9488723278045654 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7533763647079468 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6227839589118958 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3383229374885559 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_5_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6710079908370972 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +particle/sensitivity‚0.6426489949226379 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8410053253173828 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20979009568691254 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5753681659698486 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28994059562683105 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7101946473121643 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2942741811275482 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6896573305130005 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22822411358356476 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25384944677352905 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2299940288066864 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9981684684753418 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6690983772277832 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6482939720153809 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5770832896232605 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6825462579727173 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21379370987415314 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2894061505794525 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46699851751327515 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9979742765426636 )loss/val_total_loss‚7.421452397240436 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5753681659698486 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.27498453855514526 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997935950756073 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07894910126924515 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2839032709598541 *jet/accuracy_2_of_2‚0.1603655219078064 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6219704747200012 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7154925465583801 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5956888198852539 _step‚1583 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2938852310180664 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7056745290756226 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8662042021751404 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.48105159401893616 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05798707529902458 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3017432689666748 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31536251306533813 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09214159101247787 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6353245973587036 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 jet/accuracy_1_of_5‚1 jet/accuracy_4_of_5‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_2_of_3‚0.5556069612503052 +jet/accuracy_3_of_4‚0.08844424784183502 jet/accuracy_2_of_5‚1¯ ¹ßzå;‚(:ï þÛöÒõ·Ë disk.dm-1.in‚0 disk.rbd0.out‚0) disk.nvme0c0n1.out‚108551.86572265625 disk.nvme0n1.in‚ 309.37109375' disk.nvme1c1n1.out‚54105.9677734375 disk.nvme1n1.in‚ 229.87109375 proc.cpu.threads‚17) disk./.usagePercent‚76.10709534039269$ disk./.usageGB‚1168.434181213379 disk.dm-0.in‚ 539.2421875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 disk.md0.in‚0 disk.dm-0.out‚ 144934.75' disk.nvme0n1.out‚108551.86572265625* proc.memory.percent‚18.468022346496582 cpu‚25.69688051784256 network.recv‚ 4.563945e+06 disk.md0.out‚ 751.72265625# disk.dm-1.out‚16148.34423828125" disk.nvme1c1n1.in‚ 229.87109375% disk.nvme1n1.out‚54105.9677734375$ memory_percent‚97.08309173583984% proc.memory.availableMB‚ 477.90625# proc.memory.rssMB‚ 3025.80078125 network.sent‚ 3.1049708e+07¦Ý¿Ÿæ;‚(:” þÛöÒ—×ôË gpu.0.gpu‚54.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚120.116' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.038666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ç¢~ç;‚(:ó ÜöÒàÝÖË$ memory_percent‚97.08325862884521 proc.cpu.threads‚17) disk./.usagePercent‚76.10708714075031 disk.dm-0.out‚ 144935.40625 disk.rbd0.out‚0& disk.nvme0n1.out‚108553.0615234375 network.recv‚ 4.568908e+06 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚54106.7705078125( proc.memory.availableMB‚ 477.87890625# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 network.sent‚ 3.1105495e+07! disk.dm-1.out‚16148.466796875' disk.nvme1c1n1.out‚54106.7705078125 disk.md0.in‚0 disk.md0.out‚ 752.30078125 cpu‚25.696668068612016% disk./.usageGB‚1168.4340553283691 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 309.37109375( disk.nvme0c0n1.out‚108553.0615234375 disk.nvme0n1.in‚ 309.37109375ÎLCèžè;‚(:“ ÜöÒâÕÔË gpu.0.gpu‚93.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚270.214' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚90.07133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼ƒª¹4é;/ _wandbruntime‚17489  _runtime‚17489’”¤·Êê;Ä & lr-AdamW/pg1‚2.223296701464182e-06 & lr-AdamW/pg2‚2.223296701464182e-06 trainer/global_step‚19249 % _timestamp‚1.7845243163796356e+09 _runtime‚17489.955231042 _step‚1584{Öç5Ïë;É % _timestamp‚1.7845243163796356e+09 _runtime‚17489.955231042 _step‚1584 & lr-AdamW/pg1‚2.223296701464182e-06 & lr-AdamW/pg2‚2.223296701464182e-06 trainer/global_step‚19249° á²í4ì;/ _wandbruntime‚17489  _runtime‚17489‚«û«Ù í;…P ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2969922705821184 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22370441434112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22495611108864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190510179 trainer/global_step‚19249 _runtime‚17489.959151459 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33582684 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚662427716938752 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2930296200365328 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2969922705821184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45948177 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚1585 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2952632544645589 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2969922347357696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22370100009984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22495512131072 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 346424118 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845243163805354e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155913939 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2947552247347712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22336701880895 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚639932223419904 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2930296183325968 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2969922347357696 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155913939 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155913937 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚662427834528768 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22336361319621Ѳhm(î;ŠP WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _step‚1585 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2952632544645589 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2969922347357696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22370100009984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22495512131072 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 346424118 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 % _timestamp‚1.7845243163805354e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155913939 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2947552247347712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22336701880895 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚639932223419904 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2930296183325968 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2969922347357696 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155913939 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155913937 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚662427834528768 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22336361319621 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2969922705821184 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22370441434112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22495611108864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190510179 trainer/global_step‚19249 _runtime‚17489.959151459 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33582684 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚662427716938752 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2930296200365328 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2969922705821184 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45948177 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184± ×C$²ƒï;‚(:ø œÜöÒ˜‘°Ë) disk./.usagePercent‚76.10708415906217& disk.nvme1n1.out‚54107.71435546875 disk.md0.out‚ 752.44921875 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0! disk.dm-1.out‚16148.505859375( disk.nvme0c0n1.out‚108556.2451171875" disk.nvme1c1n1.in‚ 229.87109375 network.sent‚ 3.1110376e+07$ disk./.usageGB‚1168.434009552002 disk.nvme1n1.in‚ 229.87109375 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚97.08342552185059' proc.memory.availableMB‚ 477.8515625# proc.memory.rssMB‚ 3025.80078125 network.recv‚ 4.57229e+06& disk.nvme0n1.out‚108556.2451171875 cpu‚25.696471922704447 disk.md0.in‚0$ disk.dm-0.out‚144939.16162109375" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375( disk.nvme1c1n1.out‚54107.71435546875* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17lð¦üžð;‚(:“ œÜöÒ“¿àå gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚255.842' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚85.28066666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WfÛ†4ñ;/ _wandbruntime‚17490  _runtime‚17490/˜'ò;N F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33584415 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2947700521849856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45950421 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155921732 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22337810269259 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚17490.586398517 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155921732 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2970072090429440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2970071731958784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22371210108928 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79534564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2930443631032800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2970072090429440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33584418 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚662462133720576 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2952781083701313 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22496624768512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22371210108928 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190509692 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155921730 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2947700538889216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 $ _timestamp‚1.784524317010573e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 346441581 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚639965403689984 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45950149 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190519849 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22371551540224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22496730030592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155921730 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 346431422 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190519849 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2947700538889216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2930443613993440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 346431422 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344[šÈ÷#$ó;”N [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190519849 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22371551540224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22496730030592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155921730 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 346431422 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190519849 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2947700538889216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2930443613993440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 346431422 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33584415 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2947700521849856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45950421 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 155921732 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22337810269259 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _runtime‚17490.586398517 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155921732 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2970072090429440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2970071731958784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22371210108928 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79534564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2930443631032800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2970072090429440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33584418 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚662462133720576 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2952781083701313 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22496624768512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22371210108928 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190509692 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155921730 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2947700538889216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 $ _timestamp‚1.784524317010573e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 346441581 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚639965403689984 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45950149 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2954965989494272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347298420 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22391844533029 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156303587 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190984688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79731896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156303589 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190984688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2937667733738928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190994831 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22426010535936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2954965989494272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22425669111808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2977391641566720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22392185094303 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚641591236923904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22551503002624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2960059918833231 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33669381 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46062790 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156303587 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 uÔz(‚<‹P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2937667716699568 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚17535.448015571 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚664142620239360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 trainer/global_step‚19299 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156303589 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2977391641566720 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22426010535936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 % _timestamp‚1.7845243618723595e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22425669111808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚641591218311680 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 _step‚1589 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2977392000030208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2977392000030208 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2954965972454912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2954965989494272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347298420 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22391844533029 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156303587 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190984688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79731896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101074944 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156303589 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190984688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2937667733738928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190994831 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22426010535936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2954965989494272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22425669111808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2977391641566720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22392185094303 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚641591236923904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22551503002624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119687168 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2960059918833231 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33669381 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46062790 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156303587µ žä}b4ƒ</ _wandbruntime‚17536  _runtime‚17536ëÙ‹„<N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2955114246957056 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22426779210752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347315883 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156311380 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22393293482667 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347305724 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22426779210752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22392952921281 _step‚1590 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347315883 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2955114246957056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33671115 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22427120642048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2937815147367040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22552514331648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2977541384638464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347305724 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190994344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191004501 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚641624417193984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚641624398581760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22552619593728 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2955114263996416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156311380 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22427120642048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79735896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46064781 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2977541026167808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 % _timestamp‚1.7845243625042126e+09 _JDeviceSGI˜R‹tatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2937815164406400 _runtime‚17536.079998985 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191004501 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46065068 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚664176912913408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚19299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2960208457889067 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33671118 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2977541384638464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156311382 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180þBï¡'…<•N \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2955114263996416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156311380 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22427120642048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79735896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46064781 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2977541026167808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 % _timestamp‚1.7845243625042126e+09 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2937815164406400 _runtime‚17536.079998985 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191004501 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46065068 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚664176912913408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚19299 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2960208457889067 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33671118 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2977541384638464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156311382 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2955114246957056 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22426779210752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347315883 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156311380 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22393293482667 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347305724 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22426779210752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22392952921281 _step‚1590 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347315883 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2955114246957056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33671115 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22427120642048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2937815147367040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22552514331648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2977541384638464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347305724 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190994344 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191004501 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚641624417193984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚641624398581760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22552619593728 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272¶ ÃÕZ64†</ _wandbruntime‚17536  _runtime‚17536+AVð‡<‰ 0loss/FRt2/assignment_loss‚0.6833053361624479 1loss/SRqqt1/detection_loss‚0.6861829142770305 1loss/SRqqt2/detection_loss‚0.6128981991795904 0loss/FRt1/assignment_loss‚1.4365391123473756 3loss/SRqqt1/assignment_loss‚0.45499224032624547 2loss/FBt1/assignment_loss‚0.000272058898868509 2loss/FBt2/assignment_loss‚0.004676884924638262 /loss/FRt2/detection_loss‚1.5353485965645701 /loss/FBt2/detection_loss‚0.6396328266914556 % _timestamp‚1.7845243625053518e+09 _runtime‚17536.080694846 3loss/SRqqt2/assignment_loss‚0.46818990484894973 /loss/FBt1/detection_loss‚0.2551346500085926 $loss/total_loss‚7.19573725084112 epoch‚48 trainer/global_step‚19299 _step‚1591 0loss/FRt1/detection_loss‚0.41856452661135435-ñáˆ<Ž /loss/FRt2/detection_loss‚1.5353485965645701 /loss/FBt2/detection_loss‚0.6396328266914556 % _timestamp‚1.7845243625053518e+09 _runtime‚17536.080694846 3loss/SRqqt2/assignment_loss‚0.46818990484894973 /loss/FBt1/detection_loss‚0.2551346500085926 $loss/total_loss‚7.19573725084112 epoch‚48 trainer/global_step‚19299 _step‚1591 0loss/FRt1/detection_loss‚0.41856452661135435 0loss/FRt2/assignment_loss‚0.6833053361624479 1loss/SRqqt1/detection_loss‚0.6861829142770305 1loss/SRqqt2/detection_loss‚0.6128981991795904 0loss/FRt1/assignment_loss‚1.4365391123473756 3loss/SRqqt1/assignment_loss‚0.45499224032624547 2loss/FBt1/assignment_loss‚0.000272058898868509 2loss/FBt2/assignment_loss‚0.004676884924638262· M4Ú߃‰<‚(:ø ØÜöÒ§ÖóË* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.md0.out‚ 753.6640625# disk.dm-0.out‚144947.1455078125 disk.dm-1.out‚ 16148.9140625) disk.nvme0c0n1.out‚108563.74853515625 network.recv‚ 4.589659e+06) disk./.usagePercent‚76.10709633428876$ disk./.usageGB‚1168.434196472168 disk.rbd0.out‚0' disk.nvme0n1.out‚108563.74853515625" disk.nvme1c1n1.in‚ 229.87109375 cpu‚25.695585748706776 disk.dm-0.in‚ 539.2421875" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375( proc.memory.availableMB‚ 477.74609375# proc.memory.rssMB‚ 3025.80078125 network.sent‚ 3.1237101e+07& disk.nvme1n1.out‚54111.26708984375 disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme1c1n1.out‚54111.26708984375$ memory_percent‚97.08406925201416䜊<‚(:‘ ØÜöÒѳ¹Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚119.824' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.94133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&´Å}z‹<‚(:ï çÜöÒň£Ë cpu‚25.69536392122405 disk.md0.out‚ 754.015625 disk.dm-0.in‚ 539.2421875 disk.nvme0n1.in‚ 309.37109375 proc.cpu.threads‚17) disk./.usagePercent‚76.10703545815586 disk.md0.in‚0 disk.rbd0.out‚0' disk.nvme0n1.out‚108564.66650390625 disk.nvme1n1.in‚ 229.87109375# memory_percent‚97.0851182937622 network.sent‚ 3.1286723e+07 network.recv‚ 4.594294e+06 disk.dm-1.in‚0# disk.dm-1.out‚16149.02978515625" disk.nvme0c0n1.in‚ 309.37109375) disk.nvme0c0n1.out‚108564.66650390625" disk.nvme1c1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3025.80078125$ disk./.usageGB‚1168.433261871338# disk.dm-0.out‚144947.7431640625 disk.rbd0.in‚0# disk.nvme1c1n1.out‚ 54111.828125! disk.nvme1n1.out‚ 54111.828125( proc.memory.availableMB‚ 477.57421875* proc.memory.percent‚18.468022346496582æV:žŒ<‚(:“ çÜöÒÙÀþË gpu.0.gpu‚43.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚268.442' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.48066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ³ô<‚(:ô öÜöÒ¿¦ËË network.sent‚ 3.1293528e+07 network.recv‚ 4.597606e+06' disk.nvme0c0n1.out‚108567.212890625$ disk.nvme1n1.out‚54112.388671875 disk.md0.in‚0# disk.dm-1.out‚16149.07275390625 disk.rbd0.in‚0$ memory_percent‚97.08662033081055% disk./.usageGB‚1168.4332885742188" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.out‚ 754.3203125 disk.rbd0.out‚0# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 cpu‚25.695168002250092) disk./.usagePercent‚76.10703719747394" disk.nvme0c0n1.in‚ 309.37109375 disk.nvme0n1.in‚ 309.37109375% disk.nvme0n1.out‚108567.212890625 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0& proc.memory.availableMB‚ 477.328125& disk.nvme1c1n1.out‚54112.388671875$ disk.dm-0.out‚144950.13916015625 proc.cpu.threads‚17‚–&DžŽ<‚(:“ öÜöÒýæŽÌ gpu.0.gpu‚84.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚284.359' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.78633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õW·g4</ _wandbruntime‚17580  _runtime‚17580´ûíÌ<Æ trainer/global_step‚19349 % _timestamp‚1.7845244070503829e+09 _runtime‚17580.625955708 _step‚1592 ' lr-AdamW/pg1‚1.6711287569613812e-06 ' lr-AdamW/pg2‚1.6711287569613812e-06×ä†jÑ‘<Ë % _timestamp‚1.7845244070503829e+09 _runtime‚17580.625955708 _step‚1592 ' lr-AdamW/pg1‚1.6711287569613812e-06 ' lr-AdamW/pg2‚1.6711287569613812e-06 trainer/global_step‚19349¸ Žä&Á4’</ _wandbruntime‚17580  _runtime‚17580—&tI (“<…P @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2962379697562112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46177229 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2984860935775744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156693237 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79933595 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 348162577 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156693239 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚643250231815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22607352115712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚19349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191469340 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2984861294239232 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2962379714601472 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22447327746437 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22481579637760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156693237 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191469340 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22447668307711 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191479483 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191479483 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2984861294239232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 348172722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33756084 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46177511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚643250250427904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2962379697562112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2967486935420239 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2984860935775744 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚665857484953600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 _step‚1593 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2962379714601472 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2945039250073168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33756081 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156693239 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚17580.630076705 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22607253137920 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22481238213632·±j(”<ŠP `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22607253137920 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22481238213632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2962379697562112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46177229 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚98977792 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2984860935775744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156693237 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.aXÁ½×è!ll.allocated‚79933595 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 348162577 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156693239 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚643250231815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22607352115712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚19349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191469340 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2984861294239232 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2962379714601472 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22447327746437 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22481579637760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156693237 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191469340 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22447668307711 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191479483 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191479483 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2984861294239232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 348172722 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33756084 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46177511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚643250250427904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2962379697562112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2967486935420239 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2984860935775744 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚665857484953600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 _step‚1593 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2962379714601472 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2945039250073168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33756081 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156693239 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚17580.630076705 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0¹ ¼‡Ð24•</ _wandbruntime‚17581  _runtime‚17581Z?'–<N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22482689743872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191489153 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 348180026 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 348190185 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚1594 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2985010678847488 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22482689743872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22482348312576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2962527972064256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33757818 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 348180026 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156701030 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2962527989103616 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191478996 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46179868 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚643283430697984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2985010678847488 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚665891780662272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2962527989103616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46179584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2967635116875329 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚643283412085760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22608469644288 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2945186680740640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79937686 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191478996 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚19349 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156701032 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22448776696075 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2962527972064256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79937399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845244076800146e+09 _runtime‚17581.255811169 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156701032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156701030 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚665891900342272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2985010320376832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191489153 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22448436134689 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2967635116875329 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚643283412085760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22608469644288 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2945186680740640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79937686 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191478996 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚19349 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156701032 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22448776696075 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2962527972064256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79937399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 % _timestamp‚1.7845244076800146e+09 _runtime‚17581.255811169 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156701032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156701030 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚665891900342272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2985010320376832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191489153 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22448436134689 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 348190185 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 _step‚1594 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2985010678847488 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22482689743872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22482348312576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2962527972064256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33757818 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 348180026 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156701030 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2962527989103616 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191478996 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46179868 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚643283430697984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2985010678847488 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚665891780662272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2962527989103616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46179584º <5µ4˜</ _wandbruntime‚17581  _runtime‚17581HX±½™<ˆ %loss/total_loss‚6.867804499593015 epoch‚48 % _timestamp‚1.7845244076811469e+09 _step‚1595 0loss/FRt2/assignment_loss‚0.6916318261504766 /loss/FRt2/detection_loss‚1.5175957362797685 1loss/SRqqt2/detection_loss‚0.6169462495836051 1loss/SRqqt1/detection_loss‚0.6802346173012073 trainer/global_step‚19349 _runtime‚17581.25657962 /loss/FRt1/assignment_loss‚1.485066896682754 2loss/SRqqt1/assignment_loss‚0.3753796725089412 3loss/SRqqt2/assignment_loss‚0.43203095554728893 2loss/FBt1/assignment_loss‚0.020595727597963614 0loss/FRt1/detection_loss‚0.45004273317678534 2loss/FBt2/assignment_loss‚0.010261148441421284 0loss/FBt1/detection_loss‚0.31012060475597325 /loss/FBt2/detection_loss‚0.2778983315668302!°Æ€š< trainer/global_step‚19349 _runtime‚17581.25657962 /loss/FRt1/assignment_loss‚1.485066896682754 2loss/SRqqt1/assignment_loss‚0.3753796725089412 3loss/SRqqt2/assignment_loss‚0.43203095554728893 2loss/FBt1/assignment_loss‚0.020595727597963614 0loss/FRt1/detection_loss‚0.45004273317678534 2loss/FBt2/assignment_loss‚0.010261148441421284 0loss/FBt1/detection_loss‚0.31012060475597325 /loss/FBt2/detection_loss‚0.2778983315668302 %loss/total_loss‚6.867804499593015 epoch‚48 % _timestamp‚1.7845244076811469e+09 _step‚1595 0loss/FRt2/assignment_loss‚0.6916318261504766 /loss/FRt2/detection_loss‚1.5175957362797685 1loss/SRqqt2/detection_loss‚0.6169462495836051 1loss/SRqqt1/detection_loss‚0.6802346173012073» HÙº±€›<‚(:õ …ÝöÒŠåœË network.recv‚ 4.601001e+06 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚54116.2412109375# memory_percent‚97.0860481262207# proc.memory.rssMB‚ 3025.80078125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375 cpu‚25.694917714040987 proc.cpu.threads‚17& disk.nvme0n1.out‚108568.4912109375 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0# disk.dm-1.out‚16152.38427734375( disk.nvme0c0n1.out‚108568.4912109375" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚54116.2412109375 disk.md0.in‚0) disk./.usagePercent‚76.10702427682533$ disk./.usageGB‚1168.433090209961$ disk.dm-0.out‚144951.00146484375 disk.md0.out‚ 754.765625& proc.memory.availableMB‚ 477.421875* proc.memory.percent‚18.468022346496582 network.sent‚ 3.1296551e+07êè¢Õœ<‚(:’ …ÝöÒÀá’Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚282.58' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.19333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰¿NÀ‚<‚(:÷ ”ÝöÒݘÄË' disk.nvme0n1.out‚108572.06787109375% disk.nvme1c1n1.out‚54120.19140625 network.sent‚ 3.1360979e+07) disk./.usagePercent‚76.10707322620566 disk.nvme1n1.in‚ 229.87109375# disk.nvme1n1.out‚54120.19140625# disk.dm-0.out‚144954.6279296875) disk.nvme0c0n1.out‚108572.06787109375 disk.nvme0n1.in‚ 309.37109375$ memory_percent‚97.08607196807861% disk./.usageGB‚1168.4338417053223 disk.md0.out‚ 755.2109375# disk.dm-1.out‚16155.35302734375" disk.nvme0c0n1.in‚ 309.37109375* proc.memory.percent‚18.468022346496582 cpu‚25.694694155752718 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375( proc.memory.availableMB‚ 477.41796875# proc.memory.rssMB‚ 3025.80078125 proc.cpu.threads‚17 network.recv‚ 4.606335e+06 disk.md0.in‚0 disk.rbd0.out‚0ñ)Yrž<‚(:“ ”ÝöÒý¦‡Ì gpu.0.gpu‚39.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062*kŸïM, gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚286.324' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚95.44133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02\ºƒŸ<‚(:ø £ÝöÒ˜ñœË' disk.nvme0n1.out‚108573.14599609375 disk.md0.out‚ 755.66796875" disk.nvme0c0n1.in‚ 309.37109375) disk.nvme0c0n1.out‚108573.14599609375" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚54121.08447265625 disk.nvme1n1.in‚ 229.87109375$ memory_percent‚97.08728790283203# proc.memory.rssMB‚ 3025.80078125 network.recv‚ 4.609146e+06 disk.md0.in‚0& disk.nvme1n1.out‚54121.08447265625% proc.memory.availableMB‚ 477.21875 cpu‚25.694472871405925 network.sent‚ 3.1363901e+07# disk.dm-0.out‚144955.3154296875 disk.rbd0.out‚0* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17) disk./.usagePercent‚76.10706477808925% disk./.usageGB‚1168.4337120056152 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0 disk.dm-1.out‚16155.66015625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 309.37109375¿X¯[ž <‚(:“ £ÝöÒ¿ªˆÌ gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚215.266' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚71.75533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êÔ |4¡</ _wandbruntime‚17626  _runtime‚17626Ð]WÌ¢<Æ ' lr-AdamW/pg1‚1.4244758636577123e-06 ' lr-AdamW/pg2‚1.4244758636577123e-06 trainer/global_step‚19399 % _timestamp‚1.7845244525810573e+09 _runtime‚17626.157000116 _step‚1596L¥÷~Ñ£<Ë % _timestamp‚1.7845244525810573e+09 _runtime‚17626.157000116 _step‚1596 ' lr-AdamW/pg1‚1.4244758636577123e-06 ' lr-AdamW/pg2‚1.4244758636577123e-06 trainer/global_step‚19399¼ ·å.(4¤</ _wandbruntime‚17626  _runtime‚17626xBx( (¥<…P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191953992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2974913952007247 trainer/global_step‚19399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191964135 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2969793439708672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80135536 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22502810959845 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191964135 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349047024 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2974913594406613 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22503151521119 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22536807315456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2992330229984768 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33842784 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157082889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚17626.157439226 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2952410800486128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚644909245319680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157082887¡"è-(¦<ŠP ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2969793439708672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2992330588448256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33842781 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22536807315456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2992330229984768 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33842784 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157082889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚17626.157439226 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2952410800486128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚644909245319680 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157082887 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191953992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2974913952007247 trainer/global_step‚19399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191964135 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2969793439708672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80135536 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22502810959845 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191964135 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349047024 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2974913594406613 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22503151521119 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157090682 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191963648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80139248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 trainer/global_step‚19399 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22504259909483 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22503919348097 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2992479614585856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2952558231153600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2969941714210816 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2969941697171456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22538258845696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349054328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33844518 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22538258845696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157090680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191973805 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚1598 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2969941697171456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22537917414400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2952558214114240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46295018 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚667606742094848 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2975062133462337 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644942444201984 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349064487 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚667606624512000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2969941714210816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚644942425589760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80139536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33844515 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22664297892864 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2975062491063083 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22537917414400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349064487 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚667606624512000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2969941714210816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚644942425589760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80139536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33844515 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22664297892864 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2975062491063083 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22537917414400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157090682 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191963648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80139248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚98970624 trainer/global_step‚19399 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22504259909483 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22503919348097 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2992479614585856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚285 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2952558231153600 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2969941714210816 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2969941697171456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22538258845696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349054328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33844518 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22538258845696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157090680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191973805 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117582848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 _step‚1598 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2969941697171456 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22537917414400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2952558214114240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46295018 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚667606742094848 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2975062133462337 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644942444201984 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2999799524193792 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2959782333859728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22592376417280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚669287271005184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157472537 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192438644 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192438644 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46406999 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46407276 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2999799524193792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2977207164815872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192448787 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22718888300032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22558634734527 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157472539 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22592717841408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚669287147123712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚646568277435904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2982340968594255 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2959782316820368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _runtime‚17671.233718041 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157472537 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33929481 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192448787 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2999799882657280 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1601 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80336760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33929484 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚19449 % _timestamp‚1.7845244976581697e+09ªÐ²{º¸<‹P JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2999799524193792 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2959782333859728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22592376417280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚669287271005184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157472537 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192438644 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192438644 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46406999 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46407276 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2999799524193792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2977207164815872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192448787 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22718888300032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22558634734527 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157472539 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22592717841408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚669287147123712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚646568277435904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2982340968594255 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2959782316820368 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 NFDeviceSzU7WtatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _runtime‚17671.233718041 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157472537 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33929481 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192448787 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2999799882657280 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1601 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80336760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33929484 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 trainer/global_step‚19449 % _timestamp‚1.7845244976581697e+09Á ²FÂO4¹</ _wandbruntime‚17671  _runtime‚17671M@Ñ'º<N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46409267 trainer/global_step‚19449 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192448300 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚646601457705984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2982489150049345 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349928630 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22593486516224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚1602 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2977355422278656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22593486516224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚17671.845881537 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157480330 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7845244982700412e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22559402561505 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33931215 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚669321441413632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2959929747487840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157480332 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349938789 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22720111776256 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157480330 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349938789 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80340482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33931218 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192458457 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2999949267265536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349928630 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192458457 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚669321569482240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2959929764527200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80340482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33931218 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192458457 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2999949267265536 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 349928630 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192458457 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚669321569482240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2959929764527200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192448300 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚646601457705984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2982489150049345 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 349928630 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22593486516224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _step‚1602 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2977355422278656 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22593486516224 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚17671.845881537 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157480330 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7845244982700412e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22559402561505 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33931215 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚669321441413632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2959929747487840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157480332 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349938789 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22720111776256 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157480330 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349938789 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E¾<Œ 0loss/FRt1/assignment_loss‚1.4440039435515002 0loss/SRqqt2/detection_loss‚0.658099016048744 0loss/FRt2/assignment_loss‚0.6126960385299103 4loss/FBt1/assignment_loss‚3.4553222175924164e-07 3loss/FBt2/assignment_loss‚0.0001026924179329727 /loss/FRt2/detection_loss‚1.4902328410585906 /loss/FBt1/detection_loss‚0.3119935987296481 0loss/FBt2/detection_loss‚0.48294625084446224 _step‚1603 2loss/SRqqt2/assignment_loss‚0.4302575705666212 /loss/FRt1/detection_loss‚0.4862441607604136 1loss/SRqqt1/detection_loss‚0.5716037048136899 %loss/total_loss‚6.902996423791122 _runtime‚17671.84749144 2loss/SRqqt1/assignment_loss‚0.4148162609373876 epoch‚48 trainer/global_step‚19449 # _timestamp‚1.78452449827219e+09à Ʃ¥7‚¿<‚(:÷ ßÝöÒ׬ˆÌ" disk.nvme0c0n1.in‚ 309.37109375% disk./.usageGB‚1168.4338035583496 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 3025.80078125 proc.cpu.threads‚17 network.sent‚ 3.1453769e+07 disk.dm-0.in‚ 539.2421875! disk.dm-1.out‚16156.232421875) disk.nvme0c0n1.out‚108579.00146484375 disk.nvme0n1.in‚ 309.37109375% disk.nvme1c1n1.out‚54123.84765625* proc.memory.percent‚18.468022346496582 network.recv‚ 4.623903e+06) disk./.usagePercent‚76.10707074146553$ disk.dm-0.out‚144960.18310546875" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375( proc.memory.availableMB‚ 478.17578125 cpu‚25.693895879121477 disk.md0.out‚ 757.14453125' disk.nvme0n1.out‚108579.00146484375# disk.nvme1n1.out‚54123.84765625$ memory_percent‚97.08144664764404Wë’ŸÀ<‚(:” ßÝöÒ£ªÎÌ gpu.0.gpu‚43.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.682' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.227333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}žÁ<‚(:ö îÝöÒÑúÆË disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚76.10710627324922 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0# proc.memory.rssMB‚ 3025.80078125 network.sent‚ 3.1504704e+07 network.recv‚ 4.629056e+06" disk.nvme0c0n1.in‚ 309.37109375% disk.nvme0n1.out‚108580.833984375$ disk.dm-0.out‚144961.76806640625 disk.dm-1.in‚0# memory_percent‚97.0799446105957& proc.memory.availableMB‚ 478.421875% disk./.usageGB‚1168.4343490600586' disk.nvme0c0n1.out‚108580.833984375 disk.nvme0n1.in‚ 309.37109375 disk.rbd0.in‚0* proc.memory.percent‚18.468022346496582 cpu‚25.693705041603934" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚54124.51708984375& disk.nvme1n1.out‚54124.51708984375 disk.md0.out‚ 757.54296875 disk.dm-0.in‚ 539.2421875! disk.dm-1.out‚16156.302734375Òúm«žÂ<‚(:“ îÝöÒ“ý‘Ì gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚262.789' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚87.59633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]<:4Ã</ _wandbruntime‚17713  _runtime‚17713¡æIÇÄ<Á trainer/global_step‚19499 $ _timestamp‚1.784524540416424e+09 _runtime‚17713.994053711 _step‚1604 % lr-AdamW/pg1‚9.90111444200371e-07 % lr-AdamW/pg2‚9.90111444200371e-07 ÈíÌÅ<Æ trainer/global_step‚19499 $ _timestamp‚1.784524540416424e+09 _runtime‚17713.994053711 _step‚1604 % lr-AdamW/pg1‚9.90111444200371e-07 % lr-AdamW/pg2‚9.90111444200371e-07Ä ·n4Æ</ _wandbruntime‚17713  _runtime‚17713aL¥FÇ<…P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚671002146121728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34016184 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3007268818402816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22648286943232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚17713.994114811 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2984620889923072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 350795628 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.pe«Mð˜Åak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80538924 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157862187 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192933439 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2984620872883712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117590016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2967153867233328 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3007268818402816 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2984620872883712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22613777386661 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22647945519104 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192923296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46522452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚671002028531712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚648227290939904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845245404170606e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3007269176866304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22774756204032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157862187 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2984620889923072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚1605 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22774756204032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157862187 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2984620889923072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚1605 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3007268818402816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22648286943232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 _runtime‚17713.994114811 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2984620889923072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 350795628 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80538924 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157862187 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192933439 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452Å M]e4É</ _wandbruntime‚17714  _runtime‚17714’n„'Ê<N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46525007 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚671036319448576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 350802932 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3007418561474560 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2984769164425216 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2984769164425216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚17714.607002312 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22649055618048 % _timestamp‚1.7845245410312502e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22649397049344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚19499 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192932952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192932952 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80542925 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22649055618048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34017918 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2984769147385856 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2967301297900800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22775866850816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350813091 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46524721 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157869982 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚671036447517184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1606 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350813091 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 350802932 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157869980 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2989916166636353 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192943109 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22615226336299 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80542636 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34017915 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184¢€=®¾Ë<•N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚671036447517184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1606 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350813091 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 350802932 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 157869980 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2989916166636353 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192943109 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22615226336299 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80542636 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34017915 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46525007 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚671036319448576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 350802932 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3007418561474560 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2984769164425216 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚298476916442??½]5216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚289 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚17714.607002312 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22649055618048 % _timestamp‚1.7845245410312502e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22649397049344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 trainer/global_step‚19499 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 192932952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192932952 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80542925 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22649055618048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34017918 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2984769147385856 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2967301297900800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22775866850816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350813091 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46524721 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157869982 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511Æ A|z14Ì</ _wandbruntime‚17714  _runtime‚17714¨Ex Í<‹ 2loss/SRqqt2/assignment_loss‚0.4573746950796204 4loss/FBt2/assignment_loss‚2.1370057544422217e-05 1loss/SRqqt2/detection_loss‚0.6521683145742575 epoch‚48 trainer/global_step‚19499 _step‚1607 3loss/SRqqt1/assignment_loss‚0.42413495111392663 /loss/FRt2/detection_loss‚1.5186209008474287 0loss/FRt2/assignment_loss‚0.6858993663507349 /loss/FRt1/detection_loss‚0.4218201710838669 1loss/SRqqt1/detection_loss‚0.6256527670714115 $loss/total_loss‚7.09631045113928 % _timestamp‚1.7845245410322323e+09 _runtime‚17714.607497623 0loss/FRt1/assignment_loss‚1.5225866378680781 2loss/FBt1/assignment_loss‚0.002034855245397928 0loss/FBt1/detection_loss‚0.39391275074729126 0loss/FBt2/detection_loss‚0.39208367109972025ƒðaÎ< 3loss/SRqqt1/assignment_loss‚0.42413495111392663 /loss/FRt2/detection_loss‚1.5186209008474287 0loss/FRt2/assignment_loss‚0.6858993663507349 /loss/FRt1/detection_loss‚0.4218201710838669 1loss/SRqqt1/detection_loss‚0.6256527670714115 $loss/total_loss‚7.09631045113928 % _timestamp‚1.7845245410322323e+09 _runtime‚17714.607497623 0loss/FRt1/assignment_loss‚1.5225866378680781 2loss/FBt1/assignment_loss‚0.002034855245397928 0loss/FBt1/detection_loss‚0.39391275074729126 0loss/FBt2/detection_loss‚0.39208367109972025 2loss/SRqqt2/assignment_loss‚0.4573746950796204 4loss/FBt2/assignment_loss‚2.1370057544422217e-05 1loss/SRqqt2/detection_loss‚0.6521683145742575 epoch‚48 trainer/global_step‚19499 _step‚1607Ç j›Bž~Ï<‚(:ó ýÝöÒöÒ€Ì network.recv‚ 4.632506e+06 disk.dm-0.in‚ 539.2421875" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0' proc.memory.availableMB‚ 478.3828125* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17) disk./.usagePercent‚76.10710279461304 disk.dm-1.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚108581.98046875( disk.nvme1c1n1.out‚54660.28662109375& disk.nvme1n1.out‚54660.28662109375$ disk.dm-0.out‚144962.60791015625 disk.dm-1.out‚ 16691.3046875" disk.nvme0c0n1.in‚ 309.37109375& disk.nvme0c0n1.out‚108581.98046875 disk.nvme0n1.in‚ 309.37109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 758.05078125# memory_percent‚97.0801830291748 network.sent‚ 3.1520673e+07% disk./.usageGB‚1168.4342956542969 disk.rbd0.in‚0# proc.memory.rssMB‚ 3025.80078125 cpu‚25.693469294773216­‡`\‘Ð<‚(:† ýÝöÒ¼£ÀÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.293' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.431 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢lÑ<‚(:á ŒÞöÒöÈÞË% disk.nvme0c0n1.out‚108583.7265625$ memory_percent‚97.08030223846436* proc.memory.percent‚18.468022346496582 disk.nvme1n1.out‚ 54661.125! disk.dm-0.out‚144964.49609375" disk.nvme0c0n1.in‚ 309.37109375# disk.nvme0n1.out‚108583.7265625# proc.memory.rssMB‚ 3025.80078125 disk.dm-1.out‚16691.40234375) disk./.usagePercent‚75.93011972551295" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1c1n1.out‚ 54661.125 disk.md0.out‚ 758.33203125 disk.dm-0.in‚ 539.2421875 disk.dm-1.in‚0( proc.memory.availableMB‚ 478.36328125 network.sent‚ 3.1572441e+07 network.recv‚ 4.637914e+06 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375 cpu‚25.693278397665665 proc.cpu.threads‚17% disk./.usageGB‚1165.7171630859375 disk.nvme1n1.in‚ 229.87109375 disk.rbd0.in‚0Æ"Ïñ‘Ò<‚(:† ŒÞöÒÚ¢¥Ì gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚282.39' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚94.13 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WÜ~Ð|Ó<‚(:ñ ›ÞöÒð“¿Ë$ disk.nvme1n1.out‚54661.556640625 disk.dm-0.out‚ 144965.09375 disk.dm-1.in‚0) disk.nvme0c0n1.out‚108584.52099609375# proc.memory.rssMB‚ 3025.80078125 proc.cpu.threads‚17) disk./.usagePercent‚75.93012792515533% disk./.usageGB‚1165.7172889709473& disk.nvme1c1n1.out‚54661.556640625 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0 disk.dm-0.in‚ 539.2421875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.37109375 network.sent‚ 3.1577259e+07 network.recv‚ 4.641082e+06" disk.nvme1c1n1.in‚ 229.87109375# disk.dm-1.out‚16691.52001953125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 309.37109375$ memory_percent‚97.08030223846436( proc.memory.availableMB‚ 478.36328125 disk.md0.out‚758.5625' disk.nvme0n1.out‚108584.52099609375* proc.memory.percent‚18.468022346496582 cpu‚25.693045549175057 מÔ<‚(:“ ›ÞöÒô•’Ì gpu.0.gpu‚99.0 gpu.0.memory‚36, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚275.009' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.66966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©ˆ(Ó4Õ</ _wandbruntime‚17757  _runtime‚17757駱ÊÖ<Ä & lr-AdamW/pg1‚8.024278164617118e-07 & lr-AdamW/pg2‚8.024278164617118e-07 trainer/global_step‚19549 % _timestamp‚1.7845245834457607e+09 _runtime‚17757.021385041 _step‚1608çÀ.Ï×<É & lr-AdamW/pg1‚8.024278164617118e-07 & lr-AdamW/pg2‚8.024278164617118e-07 trainer/global_step‚19549 % _timestamp‚1.7845245834457607e+09 _runtime‚17757.021385041 _step‚1608È ):M`4Ø</ _wandbruntime‚17757  _runtime‚17757é4 (Ù<†P OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34102881 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚19549 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193418091 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22830570347008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34102884 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193407948 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2992034597990912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 _step‚1609 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3014738112611840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚672716977963008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22830673519104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚17757.022244042 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2992034597990912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22703514620928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158251837 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158251839 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2992034615030272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2997195001768271 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 351659785 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3014738471075328 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46637121 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649886304443904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845245834467587e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158251837 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2992034615030272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 351669930 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3014738471075328 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22703856045056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46637401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22703856045056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193418091æ€`(Ú<‹P [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22669260600069 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193407948 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚672716856178688 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22669601161343 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2974525383567568 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22703514620928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80740285 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚649886285831680 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2997194644167637 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3014738112611840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚672716977963008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22830673519104 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚17757.022244042 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2992034597990912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22703514620928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158251837 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158251839 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2992034615030272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2997195001768271 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 351659785 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3014738471075328 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46637121 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚649886304443904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 % _timestamp‚1.7845245834467587e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158251837 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2992034615030272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 351669930 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3014738471075328 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22703856045056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46637401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22703856045056 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193418091 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34102881 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚19549 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193418091 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚22830570347008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34102884 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193407948 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2992034597990912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 _step‚1609 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚649919484713984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351687393 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46639401 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2992182889532416 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2992182889532416 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2992182872493056 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22704624719872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80744299 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚17757.633350657 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitorä½Q¢Ä.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2997343183223361 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193427761 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2974672814235040 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22670368988321 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚672751273639936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3014887855683584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22704624719872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3014887497212928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80744016 _step‚1610 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193417604 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22704966151168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158259630 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158259632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158259630 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚649919466101760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158259632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34104615 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193427761 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193427761 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚649919484713984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 351687393 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46639401 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2992182889532416 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2992182889532416 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2992182872493056 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22704624719872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80744299 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 _runtime‚17757.633350657 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119680000 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2997343183223361 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193427761 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2974672814235040 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22670368988321 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚672751273639936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101067776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3014887855683584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22704624719872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3014887497212928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80744016 _step‚1610 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193417604 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22704966151168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158259630 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158259632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158259630 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚649919466101760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158259632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34104615 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158641487 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 352534087 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158641487 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80941828 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34189584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚651545317947904 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22759083722752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚674431853634048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22886535686144 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3004622018355279 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚651545299335680 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2999448340137472 _step‚1613 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193902743 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2981896933980528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22724743813477 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3022207406820864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158641489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2999448323098112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2999448323098112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193892600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3022207406820864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22759425146880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3022207765284352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2999448340137472Dí(ì<‹P VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 130172928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193902743 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2981896933980528 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22724743813477 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3022207406820864 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158641489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2999448323098112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2999448323098112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193892600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3022207406820864 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22759425146880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3022207765284352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2999448340137472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚674431723461120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22725084374751 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46752247 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7845246264158401e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158641487 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 352534087 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158641487 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80941828 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34189584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚651545317947904 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22759083722752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚674431853634048 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22886535686144 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3004622018355279 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚651545299335680 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2999448340137472 _step‚1613Í ²…W¶4í</ _wandbruntime‚17800  _runtime‚17800Ä5…÷'î<ŽN L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193912413 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80945841 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193902256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 352561695 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22725852201729 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2999596597600256 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2999596614639616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3004770557411115 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2982044347608640 _runtime‚ 17800.6014522 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80946130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34191318 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46754812 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22887540318208 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22726192763115 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2999596597600256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34191315 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22760193821696 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2982044364648000 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158649280 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3022357149892608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158649282 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352551536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651578479605760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352561695 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158649280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3022356791421952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚674466019923968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22760535252992 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352561695 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158649280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3022356791421952 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚674466019923968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22760535252992 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193912413 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80945841 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193902256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 352561695 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22725852201729 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2999596597600256 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2999596614639616 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3004770557411115 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2982044347608640 _runtime‚ 17800.6014522 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80946130 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34191318 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46754812 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22887540318208 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22726192763115 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2999596597600256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34191315 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22760193821696 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2982044364648000 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158649280 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3022357149892608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158649282 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352551536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651578479605760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 E&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *particle/specificity‚0.910741925239563 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30247747898101807 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9096983075141907 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 trainer/global_step‚19599 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5965873599052429 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22810806334018707 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09225203096866608 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2743787169456482 +jet/accuracy_3_of_3‚0.07523223012685776 % _timestamp‚1.7845246430096698e+09 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5708333253860474 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_1‚0.3309534788131714 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34747156500816345 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2391413450241089 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_2‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30013221502304077 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7286843061447144 particle/accuracy_2_of_4‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6815326809883118 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7058231234550476 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6451610326766968 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07116943597793579 *jet/accuracy_1_of_4‚0.9695299863815308 particle/accuracy_1_of_3‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2898847162723541 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5823055505752563 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2937626540660858 particle/accuracy_5_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2807809114456177 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21291068196296692 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6954892873764038 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 _step‚1616 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997935950756073 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6239684820175171 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03946898505091667 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5781670212745667 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6886273622512817 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27005496621131897 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5961230397224426 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5761319398880005 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21827223896980286 (particle/accuracy‚0.8530815839767456 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5739886164665222 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07834848761558533 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6158114671707153 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28929391503334045 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21795658767223358 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33837300539016724 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5652887225151062 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29457440972328186 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7733877301216125 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2749095857143402 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08074092119932175 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 /particle/accuracy_4_of_4‚0.9539412260055542 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 jet/accuracy_3_of_5‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7193347215652466 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8410053253173828 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4464632570743561 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47709497809410095 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28929391503334045 particle/accuracy_4_of_5‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6376804709434509 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9942034482955933 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6687403917312622 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20891886949539185 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26425138115882874 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07992091774940491 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.620489776134491 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7241690754890442 *jet/accuracy_2_of_4‚0.7320336699485779 particle/accuracy_3_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 )loss/val_total_loss‚7.424616904953633 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07781626284122467 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3002336025238037 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22999322414398193 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7077344655990601 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31566569209098816 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.714832603931427 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861111044883728 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04760538786649704 +jet/accuracy_3_of_4‚0.08844424784183502 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6472066640853882 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6467363834381104 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46497127413749695 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 epoch‚48 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6226282715797424 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9931437373161316 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5984963178634644 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5657804608345032 7"Purity/1FRt*SRqqt1FBt/event_purity‚0.50638347864151 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2543603777885437 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05044812336564064 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8641345500946045 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5960505604743958 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 &particle/f_score‚0.652475118637085 particle/accuracy_1_of_5‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11264298111200333 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3454256057739258 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6223734617233276 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07116943597793579 particle/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9488723278045654 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22718767821788788 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9981684684753418 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7733877301216125 *jet/accuracy_1_of_2‚0.7714372277259827 *jet/accuracy_2_of_3‚0.5551445484161377 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6656720638275146 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4799939692020416 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9994120597839355 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8866800665855408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9975854754447937 *jet/accuracy_2_of_2‚0.1600373536348343 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3016418516635895 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2839032709598541 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5776960253715515 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 +particle/sensitivity‚0.6422553062438965 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6690449714660645 particle/accuracy_1_of_2‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.622687041759491 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.294146865606308 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9979742765426636 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 *jet/accuracy_1_of_3‚0.9182965159416199 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26319369673728943 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5761319398880005 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6461771130561829 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7validation_average_jet_accuracy‚0.37176188826560974 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.866129994392395 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6354772448539734 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47899165749549866 _runtime‚17816.58580199 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7528538107872009 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3431586027145386 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8389833569526672 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6593635678291321 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05796406418085098 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 jet/accuracy_4_of_5‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7095342874526978 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ä|Q:÷<¤z 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.622687041759491 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.294146865606308 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9979742765426636 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 *jet/accuracy_1_of_3‚0.9182965159416199 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26319369673728943 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5761319398880005 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6461771130561829 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7validation_average_jet_accuracy‚0.37176188826560974 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.866129994392395 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6354772448539734 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47899165749549866 _runtime‚17816.58580199 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7528538107872009 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3431586027145386 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8389833569526672 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6593635678291321 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05796406418085098 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 jet/accuracy_4_of_5‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7095342874526978 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *particle/specificity‚0.910741925239563 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30247747898101807 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9096983075141907 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 trainer/global_step‚19599 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5965873599052429 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22810806334018707 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09225203096866608 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2743787169456482 +jet/accuracy_3_of_3‚0.07523223012685776 % _timestamp‚1.7845246430096698e+09 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5708333253860474 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_1‚0.3309534788131714 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34747156500816345 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2391413450241089 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_2‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.30013221502304077 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7286843061447144 particle/accuracy_2_of_4‚1 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6815326809883118 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7058231234550476 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6451610326766968 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07116943597793579 *jet/accuracy_1_of_4‚0.9695299863815308 particle/accuracy_1_of_3‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2898847162723541 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5823055505752563 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2937626540660858 particle/accuracy_5_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2807809114456177 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21291068196296692 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6954892873764038 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 _step‚1616 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997935950756073 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6239684820175171 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03946898505091667 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5781670212745667 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6886273622512817 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27005496621131897 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5961230397224426 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5761319398880005 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21827223896980286 (particle/accuracy‚0.8530815839767456 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5739886164665222 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07834848761558533 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6158114671707153 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28929391503334045 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21795658767223358 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33837300539016724 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5652887225151062 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.29457440972328186 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7733877301216125 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2749095857143402 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08074092119932175 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 /particle/accuracy_4_of_4‚0.9539412260055542 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 jet/accuracy_3_of_5‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7193347215652466 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8410053253173828 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4464632570743561 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47709497809410095 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28929391503334045 particle/accuracy_4_of_5‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6376804709434509 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9942034482955933 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6687403917312622 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20891886949539185 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.26425138115882874 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07992091774940491 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.620489776134491 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7241690754890442 *jet/accuracy_2_of_4‚0.7320336699485779 particle/accuracy_3_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 )loss/val_total_loss‚7.424616904953633 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07781626284122467 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3002336025238037 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22999322414398193 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7077344655990601 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31566569209098816 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.714832603931427 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861111044883728 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04760538786649704 +jet/accuracy_3_of_4‚0.08844424784183502 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6472066640853882 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6467363834381104 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46497127413749695 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 epoch‚48 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6226282715797424 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9931437373161316 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5984963178634644 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5657804608345032 7"Purity/1FRt*SRqqt1FBt/event_purity‚0.50638347864151 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2543603777885437 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05044812336564064 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8641345500946045 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5960505604743958 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 &particle/f_score‚0.652475118637085 particle/accuracy_1_of_5‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11264298111200333 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3454256057739258 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6223734617233276 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07116943597793579 particle/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9488723278045654 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.22718767821788788 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9981684684753418 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,jet/accuracy_4_of_4‚0.013899499550461769 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7733877301216125 *jet/accuracy_1_of_2‚0.7714372277259827 *jet/accuracy_2_of_3‚0.5551445484161377 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6656720638275146 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4799939692020416 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9994120597839355 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8866800665855408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9975854754447937 *jet/accuracy_2_of_2‚0.1600373536348343 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3016418516635895 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚å(öÙ1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2839032709598541 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5776960253715515 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 +particle/sensitivity‚0.6422553062438965 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6690449714660645 particle/accuracy_1_of_2‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936Ð ·w>‡ø<‚(:ô æÞöÒ‚âÔË! disk.nvme0c0n1.in‚ 309.6328125" disk.nvme1c1n1.in‚ 229.87109375% proc.memory.availableMB‚ 477.96875* proc.memory.percent‚18.468022346496582 network.recv‚ 4.661889e+06) disk./.usagePercent‚76.19693584035262% disk./.usageGB‚1169.8134574890137 disk.dm-1.in‚0" disk.dm-1.out‚16692.2451171875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 309.6328125$ disk.nvme1n1.out‚54671.759765625 disk.rbd0.in‚0) disk.nvme0c0n1.out‚111262.76123046875& disk.nvme1c1n1.out‚54671.759765625 disk.md0.in‚0 disk.md0.out‚ 762.421875$ disk.dm-0.out‚147644.77001953125# proc.memory.rssMB‚ 3025.80078125 cpu‚25.694405830976372' disk.nvme0n1.out‚111262.76123046875 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 539.50390625$ memory_percent‚97.08271026611328 proc.cpu.threads‚17 network.sent‚ 3.1715046e+07¶X„žù<‚(:“ æÞöÒõÊêË gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚280.936' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.64533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Í’€ú<‚(:õ õÞöÒø‹ÓË disk.nvme1n1.in‚ 229.87109375$ disk./.usageGB‚1169.766315460205 disk.md0.in‚0# disk.dm-1.out‚16692.66455078125 disk.rbd0.in‚0" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚54789.56396484375& disk.nvme1n1.out‚54789.56396484375* proc.memory.percent‚18.468022346496582) disk./.usagePercent‚76.19386519851523 disk.md0.out‚763.375 disk.dm-0.in‚ 543.7109375" disk.nvme0c0n1.in‚ 313.83984375 disk.nvme0n1.in‚ 313.83984375% proc.memory.availableMB‚ 478.21875# proc.memory.rssMB‚ 3025.80078125 cpu‚25.694158482117142 disk.dm-1.in‚0$ memory_percent‚97.08118438720703 proc.cpu.threads‚17) disk.nvme0c0n1.out‚112635.90087890625 network.sent‚ 3.1773976e+07 network.recv‚ 4.66661e+06$ disk.dm-0.out‚149133.29833984375 disk.rbd0.out‚0' disk.nvme0n1.out‚112635.90087890625‡®žû<‚(:“ õÞöÒ»èÌ gpu.0.gpu‚30.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.638' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.21266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KÆs€ü<‚(:õ „ßöÒë¶ÃË) disk./.usagePercent‚76.19379786205805 disk.md0.out‚ 763.93359375 disk.dm-0.in‚ 543.72265625 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 313.8515625# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582% disk.nvme1c1n1.out‚54791.49609375 disk.md0.in‚0" disk.dm-0.out‚149144.337890625# disk.dm-1.out‚16692.69580078125) disk.nvme0c0n1.out‚112646.21630859375 cpu‚25.693925884560393$ disk./.usageGB‚1169.765281677246" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375# disk.nvme1n1.out‚54791.49609375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 313.8515625 network.sent‚ 3.1776898e+07 disk.rbd0.in‚0' disk.nvme0n1.out‚112646.21630859375$ memory_percent‚97.08120822906494( proc.memory.availableMB‚ 478.21484375 proc.cpu.threads‚17 network.recv‚ 4.669939e+06~¨|ïžý<‚(:“ „ßöÒ¦àšÌ gpu.0.gpu‚97.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚278.555' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.85166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t§›Ó4þ</ _wandbruntime‚17859  _runtime‚17859ýBÊÿ<Ä & lr-AdamW/pg1‚4.861179970336688e-07 & lr-AdamW/pg2‚4.861179970336688e-07 trainer/global_step‚19649 % _timestamp‚1.7845246855757604e+09 _runtime‚17859.151392938 _step‚1617¹AÄÏ€=É & lr-AdamW/pg1‚4.861179970336688e-07 & lr-AdamW/pg2‚4.861179970336688e-07 trainer/global_step‚19649 % _timestamp‚1.7845246855757604e+09 _runtime‚17859.151392938 _step‚1617Ñ ƒèÂ4=/ _wandbruntime‚17859  _runtime‚17859æÅ(‚ (‚=…P F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 trainer/global_step‚19649 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34280397 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3031639721204736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 353622836 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22835312084480 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 194458746 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3031639362741248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3008804409120256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚676189894665728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353612691 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194458746 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚676190008061440 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81182507 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3008804392080896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22800530875782 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46902384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7845246855767496e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81182784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3008804409120256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180½#Qp(ƒ=ŠP KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚22963097947136 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3008804392080896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46902110 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚653226891502080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3013990237198774 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22835312084480 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚653226910114304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94783488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194468889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22800871437056 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22834970660352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22835312084480 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 194458746 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3031639362741248 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3008804409120256 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚676189894665728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353612691 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194458746 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚676190008061440 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81182507 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3008804392080896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22800530875782 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46902384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7845246855767496e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81182784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3008804409120256 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 trainer/global_step‚19649 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34280397 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3031639721204736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 353622836 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353640299 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159161740 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159161738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 353630140 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚676224185623552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚19649 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22836080759296 _runtime‚17859.771386358 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3031788747342336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22836080759296 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_p}5žÑ ool.allocated‚ 194478559 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81186507 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194478559 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3008952683622400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3008952666583040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81186801 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159161738 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚653260090384384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22836422190592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 % _timestamp‚1.7845246861956515e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22964210724864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0§°©Û'†=”N D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3014138776254498 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22836422190592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚22801979825420 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3031789105812992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46904667 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 trainer/global_step‚19649 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚22836080759296 _runtime‚17859.771386358 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3031788747342336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22836080759296 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194478559 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81186507 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194478559 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3008952683622400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3008952666583040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81186801 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159161738 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115485696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚653260090384384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22836422190592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 % _timestamp‚1.7845246861956515e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚294 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚291 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚22964210724864 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2991337154029824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159161740 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚22964113851392 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3014139133855244 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353640299 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159161740 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159161738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96873472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 353630140 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚676224185623552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3039108656950272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81383931 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159543595 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3016218134227456 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3016218117188096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194943398 trainer/global_step‚19699 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2998561256735952 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 354486993 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22890881186304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3021417253785782 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81383651 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22890539762176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚677904752843776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚654885923618304 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3039109015413760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194953541 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚654885905006080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3016218134227456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22890881186304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47016554 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159543597 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159543595 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2998561239696592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799¦¬Ð» •=‹P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159543595 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3016218134227456 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3016218117188096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194943398 trainer/global_step‚19699 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2998561256735952 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 354486993 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22890881186304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3021417253785782 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81383651 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_s,ÅÑBVtart/allocated_bytes.small_pool.current‚ 341424128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22890539762176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚677904752843776 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚654885923618304 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3039109015413760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194953541 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚654885905006080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3016218134227456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚22890881186304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47016554 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159543597 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159543595 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2998561239696592 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23018847837696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 194953541 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3021417611386416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47016831 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3039108656950272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81383931Ö 9è¡©4–=/ _wandbruntime‚17902  _runtime‚17902eü%d'—=N D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159551390 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3021566150442252 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22857122477442 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 354504442 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚17902.988497813 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3039258400022016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚677939170829824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚654919103888384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159551388 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23020066941440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 354514601 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3016366408729600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚19699 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81387705 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34368834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3039258041551360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354514601 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23019959582208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845247294126468e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22891991292416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3039258400022016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194963211 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159551390 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3021565792841506 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚1623 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47018874 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81388003 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2998708670364064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194953054 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3016366408729600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194963211 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159551390 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3021565792841506 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚1623 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47018874 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81388003 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2998708670364064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194953054 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3016366408729600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159551390 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3021566150442252 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22857122477442 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 354504442 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 _runtime‚17902.988497813 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3039258400022016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚677939170829824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚654919103888384 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159551388 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚295 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23020066941440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 354514601 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3016366408729600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 trainer/global_step‚19699 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81387705 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34368834 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 107359232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3039258041551360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 125971456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354514601 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23019959582208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845247294126468e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22891991292416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452× Þ»ÈU4™=/ _wandbruntime‚17902  _runtime‚17902µšVš=Œ _step‚1624 0loss/FBt2/detection_loss‚0.48877261315681697 epoch‚49 trainer/global_step‚19699 % _timestamp‚1.7845247294136317e+09 2loss/SRqqt1/assignment_loss‚0.4459316206310273 4loss/FBt2/assignment_loss‚1.0191348744047532e-05 _runtime‚17902.988992744 /loss/FRt1/assignment_loss‚1.469172381446475 0loss/FRt2/assignment_loss‚0.5819362142109519 2loss/SRqqt2/assignment_loss‚0.4930826845653191 /loss/FRt1/detection_loss‚0.4676828697096928 1loss/SRqqt1/detection_loss‚0.6005392905582037 1loss/SRqqt2/detection_loss‚0.5846679462640209 /loss/FBt1/detection_loss‚0.4884659129840099 4loss/FBt1/assignment_loss‚0.00019611302645760565 /loss/FRt2/detection_loss‚1.5146225854390865 &loss/total_loss‚7.1350804233408045‡éj¦›=‘ 2loss/SRqqt2/assignment_loss‚0.4930826845653191 /loss/FRt1/detection_loss‚0.4676828697096928 1loss/SRqqt1/detection_loss‚0.6005392905582037 1loss/SRqqt2/detection_loss‚0.5846679462640209 /loss/FBt1/detection_loss‚0.4884659129840099 4loss/FBt1/assignment_loss‚0.00019611302645760565 /loss/FRt2/detection_loss‚1.5146225854390865 &loss/total_loss‚7.1350804233408045 _step‚1624 0loss/FBt2/detection_loss‚0.48877261315681697 epoch‚49 trainer/global_step‚19699 % _timestamp‚1.7845247294136317e+09 2loss/SRqqt1/assignment_loss‚0.4459316206310273 4loss/FBt2/assignment_loss‚1.0191348744047532e-05 _runtime‚17902.988992744 /loss/FRt1/assignment_loss‚1.469172381446475 0loss/FRt2/assignment_loss‚0.5819362142109519Ø n ˜}œ=‚(:ò ÀßöÒï½êË* proc.memory.percent‚18.468022346496582 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 313.8515625) disk.nvme0c0n1.out‚112666.07666015625 disk.nvme0n1.in‚ 313.8515625" disk.nvme1c1n1.in‚ 229.87109375$ disk.nvme1c1n1.out‚ 54795.3046875# proc.memory.rssMB‚ 3025.80078125 network.sent‚ 3.1866762e+07 network.recv‚ 4.68525e+06 disk.nvme1n1.in‚ 229.87109375# disk.dm-0.out‚149163.8662109375 disk.dm-1.in‚0$ memory_percent‚97.08619117736816 cpu‚25.693080638700824" disk.nvme1n1.out‚ 54795.3046875% disk./.usageGB‚1167.0480995178223' disk.nvme0n1.out‚112666.07666015625 disk.md0.out‚ 765.6015625# disk.dm-1.out‚16693.26611328125' proc.memory.availableMB‚ 477.3984375 proc.cpu.threads‚17) disk./.usagePercent‚76.01681156279578 disk.rbd0.in‚0 disk.md0.in‚0 disk.dm-0.in‚ 543.72265625¯ ×Ï“=‚(:ˆ ÀßöÒǵ¯Ì gpu.0.gpu‚92.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚280.707' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚93.569 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£j˜…‚ž=‚(:÷ ÏßöÒšõÄË) disk./.usagePercent‚76.01677752185617% disk./.usageGB‚1167.0475769042969" disk.dm-1.out‚16693.3681640625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 313.8515625$ disk.nvme0n1.out‚112667.67578125 disk.md0.out‚ 765.88671875 disk.dm-0.in‚ 543.72265625 network.sent‚ 3.1918376e+07 network.recv‚ 4.690309e+06& disk.nvme0c0n1.out‚112667.67578125( disk.nvme1c1n1.out‚54795.81787109375& disk.nvme1n1.out‚54795.81787109375 disk.md0.in‚0$ disk.dm-0.out‚149165.25732421875$ memory_percent‚97.08669185638428" disk.nvme1c1n1.in‚ 229.87109375 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 313.8515625( proc.memory.availableMB‚ 477.31640625# proc.memory.rssMB‚ 3025.80078125 disk.dm-1.in‚0 disk.nvme1n1.in‚ 229.87109375* proc.memory.percent‚18.468022346496582 cpu‚25.692892221044765 proc.cpu.threads‚17-_ûžŸ=‚(:“ ÏßöÒ…‚þË gpu.0.gpu‚71.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚269.155' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚89.71833333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù=Œ§‚ =‚(:÷ ÞßöÒ°»¬Ë network.recv‚ 4.693836e+06 disk.md0.out‚ 766.23046875 disk.dm-0.in‚ 543.72265625 disk.dm-1.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 313.8515625* proc.memory.percent‚18.468022346496582 cpu‚25.692689893933945% disk./.usageGB‚1167.0475959777832" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚54796.88232421875 disk.md0.in‚0" disk.dm-1.out‚16693.4384765625$ memory_percent‚97.08671569824219# proc.memory.rssMB‚ 3025.80078125) disk./.usagePercent‚76.01677876422623' disk.nvme0n1.out‚112677.82861328125( disk.nvme1c1n1.out‚54796.88232421875$ proc.memory.availableMB‚477.3125 network.sent‚ 3.1923259e+07 disk.nvme0n1.in‚ 313.8515625" disk.dm-0.out‚149175.673828125 disk.rbd0.in‚0) disk.nvme0c0n1.out‚112677.82861328125 proc.cpu.threads‚17ñœ“¡=‚(:ˆ ÞßöÒ«ŠïË gpu.0.gpu‚53.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚121.443' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.481 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mêÆ4¢=/ _wandbruntime‚17945  _runtime‚17945zF£=ÄݬpÄ & lr-AdamW/pg1‚2.486170369466456e-07 & lr-AdamW/pg2‚2.486170369466456e-07 trainer/global_step‚19749 % _timestamp‚1.7845247716189702e+09 _runtime‚17945.196040047 _step‚1625[Ü+)Ϥ=É & lr-AdamW/pg1‚2.486170369466456e-07 & lr-AdamW/pg2‚2.486170369466456e-07 trainer/global_step‚19749 % _timestamp‚1.7845247716189702e+09 _runtime‚17945.196040047 _step‚1625Ù ?½ÁÝ4¥=/ _wandbruntime‚17945  _runtime‚17945ñïèì (¦=…P QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81585019 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚679619722982400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3023631842295296 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3046577951159296 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3023631842295296 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22911837863872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚19749 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159933247 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159933245 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47131219 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3028844270372790 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3046578309622784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81584741 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚679619599100928 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3005932773070192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22946108864000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355361295 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195438193 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3028844627973424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚17945.196117037 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3023631859334656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23074680590848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195428050 _step‚1626 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195438193 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚656544937122304 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195428050 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22946108864000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159933247 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47130944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22946450288128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303Ð…/l(§=ŠP ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚679619599100928 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3005932773070192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚22946108864000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355361295 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195438193 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3028844627973424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 _runtime‚17945.196117037 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3023631859334656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23074680590848 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195428050 _step‚1626 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195438193 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚656544937122304 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195428050 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚22946108864000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159933247 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47130944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚22946450288128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81585019 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚679619722982400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3023631842295296 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3046577951159296 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3023631842295296 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚22911837863872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚19749 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159933247 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159933245 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47131219 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3028844270372790 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3046578309622784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81584741Ú ¿¤n4¨=/ _wandbruntime‚17945  _runtime‚17945½ÌuÇ'©=N UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚19749 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195437706 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22912605690850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34455531 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22947218962944 _runtime‚17945.808751865 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81589014 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23075790631424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784524772232992e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3046727694231040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159941038 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195447863 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22947560394240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159941040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355378744 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355388903 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81588727 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3028993167029260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3023780116797440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195437706 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3046727694231040 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3023780133836800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159941040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3023780133836800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3006080220777024 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3023780116797440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚679654019577344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3028992809428514 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22947560394240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195447863 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34455534 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚656578098780160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1627 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0)ŸÌŸª=”N _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3023780133836800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159941040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3023780133836800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3006080220777024 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3023780116797440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚679654019577344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 130165760 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3028992809428514 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E?DeviceStatsMonitor.A™} .on_train_batch_end/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚22947560394240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195447863 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34455534 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚656578098780160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1627 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚19749 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195437706 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22912605690850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34455531 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚22947218962944 _runtime‚17945.808751865 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81589014 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23075790631424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784524772232992e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3046727694231040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159941038 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195447863 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚22947560394240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159941040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355378744 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355388903 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81588727 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3028993167029260 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3023780116797440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195437706 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3046727694231040Û K–ë4«=/ _wandbruntime‚17945  _runtime‚17945ifø.¬=ˆ 2loss/SRqqt2/assignment_loss‚0.4650213774454291 0loss/FBt2/detection_loss‚0.35986550292097186 trainer/global_step‚19749 $ _timestamp‚1.784524772233981e+09 /loss/FRt1/assignment_loss‚1.414134656086969 0loss/FRt2/assignment_loss‚0.6747082095541712 3loss/SRqqt1/assignment_loss‚0.42852760601703527 3loss/FBt1/assignment_loss‚0.0036401640385443643 2loss/FBt2/assignment_loss‚9.75532537170112e-05 0loss/FRt1/detection_loss‚0.44970824504107626 /loss/FRt2/detection_loss‚1.4246548223212654 0loss/SRqqt1/detection_loss‚0.640467597892774 /loss/FBt1/detection_loss‚0.2716020870291025 %loss/total_loss‚6.718476451356092 epoch‚49 _runtime‚17945.809353016 _step‚1628 1loss/SRqqt2/detection_loss‚0.5860486297550362)ÑÚv­= /loss/FRt2/detection_loss‚1.4246548223212654 0loss/SRqqt1/detection_loss‚0.640467597892774 /loss/FBt1/detection_loss‚0.2716020870291025 %loss/total_loss‚6.718476451356092 epoch‚49 _runtime‚17945.809353016 _step‚1628 1loss/SRqqt2/detection_loss‚0.5860486297550362 2loss/SRqqt2/assignment_loss‚0.4650213774454291 0loss/FBt2/detection_loss‚0.35986550292097186 trainer/global_step‚19749 $ _timestamp‚1.784524772233981e+09 /loss/FRt1/assignment_loss‚1.414134656086969 0loss/FRt2/assignment_loss‚0.6747082095541712 3loss/SRqqt1/assignment_loss‚0.42852760601703527 3loss/FBt1/assignment_loss‚0.0036401640385443643 2loss/FBt2/assignment_loss‚9.75532537170112e-05 0loss/FRt1/detection_loss‚0.44970824504107626Ü áÒ;ê®=‚(:ö íßöÒÙ¨Ë' disk.nvme0n1.out‚112678.63525390625( disk.nvme1c1n1.out‚54797.40966796875 network.recv‚ 4.697312e+06) disk./.usagePercent‚76.01679292724491 disk.nvme1n1.in‚ 229.87109375 disk.md0.in‚0" disk.nvme1c1n1.in‚ 229.87109375$ proc.memory.availableMB‚476.8125# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582& disk.nvme1n1.out‚54797.40966796875" disk.dm-0.out‚149176.201171875 disk.dm-1.in‚0" disk.dm-1.out‚16693.6005859375$ memory_percent‚97.08976745605469 disk.md0.out‚ 766.5234375 disk.dm-0.in‚ 543.72265625 disk.nvme0n1.in‚ 313.8515625 cpu‚25.692529149014767 proc.cpu.threads‚17 network.sent‚ 3.1926181e+07% disk./.usageGB‚1167.0478134155273 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 313.8515625) disk.nvme0c0n1.out‚112678.63525390625?=Ÿž¯=‚(:“ íßöÒ¡Ö´Ì gpu.0.gpu‚54.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚283.934' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚94.64466666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§dån…°=‚(:ú üßöÒ£ô¬Ë' disk.nvme0n1.out‚113686.91748046875 disk.md0.out‚ 766.73046875 disk.nvme0n1.in‚ 313.8515625 network.sent‚ 3.1989928e+07( disk./.usagePercent‚76.1842005533536" disk.dm-1.out‚16693.6357421875" disk.nvme1c1n1.in‚ 229.87109375( disk.nvme1c1n1.out‚54797.78564453125 disk.md0.in‚0( proc.memory.availableMB‚ 477.05859375" disk.dm-0.out‚150178.111328125 disk.dm-1.in‚0& disk.nvme1n1.out‚54797.78564453125$ memory_percent‚97.08826541900635# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 cpu‚25.692354582721958 network.recv‚ 4.701838e+06% disk./.usageGB‚1169.6179389953613 disk.dm-0.in‚ 543.72265625 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 313.8515625 disk.nvme1n1.in‚ 229.87109375 proc.cpu.threads‚17 disk.rbd0.out‚0) disk.nvme0c0n1.out‚113686.91748046875­ YgŸ±=‚(:” üßöÒÑÌûË gpu.0.gpu‚29.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.313' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.437666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²r|²=‚(:ñ ‹àöÒà·ÖË disk.dm-1.in‚0 disk.rbd0.out‚0" disk.nvme1c1n1.in‚ 229.87109375' disk.nvme1c1n1.out‚54798.3662109375% disk.nvme1n1.out‚54798.3662109375 disk.md0.in‚0 network.sent‚ 3.1994816e+07 network.recv‚ 4.70533e+06 disk.nvme0n1.in‚ 313.8515625 disk.md0.out‚ 767.109375# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 cpu‚25.692568301341716 disk.dm-0.in‚ 543.72265625" disk.dm-0.out‚151092.107421875 disk.rbd0.in‚0% disk.nvme0n1.out‚114595.017578125 disk.nvme1n1.in‚ 229.87109375$ memory_percent‚97.08828926086426' proc.memory.availableMB‚ 477.0546875) disk./.usagePercent‚76.19671618932622" disk.dm-1.out‚16693.7451171875! disk.nvme0c0n1.in‚ 313.8515625' disk.nvme0c0n1.out‚114595.017578125 proc.cpu.threads‚17% disk./.usageGB‚1169.8100852966309EÀž³=‚(:“ ‹àöÒœæ˜Ì gpu.0.gpu‚57.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚278.314' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚92.77133333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µG<4´=/ _wandbruntime‚17988  _runtime‚179882`Èʵ=Ä _runtime‚17988.443451928 _step‚1629 & lr-AdamW/pg1‚1.594397380027137e-07 & lr-AdamW/pg2‚1.594397380027137e-07 trainer/global_step‚19799 % _timestamp‚1.7845248148678615e+09Þ©™+϶=É _step‚1629 & lr-AdamW/pg1‚1.594397380027137e-07 & lr-AdamW/pg2‚1.594397380027137e-07 trainer/global_step‚19799 % _timestamp‚1.7845248148678615e+09 _runtime‚17988.443451928Ý AU4·=/ _wandbruntime‚17988  _runtime‚17988±Ï*¥ (¸=„P WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3054047603831808 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3031045567402496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚658203950626304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 356245742 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195922845 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3031045584441856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚681334521979904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81786465 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3013304306443792 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23001677965824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47245677 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3031045567402496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚658203932014080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚22966980516006 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚681334393904128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 160322897 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23130461890048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160322895 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34540500 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23130571353600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 195912702 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81786174 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 _runtime‚17988.44429553 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34540497 $ _timestamp‚1.784524814868826e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23002019389952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23002019389952 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3054047603831808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3054047245368320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3013304323483152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _step‚1630 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34540497 $ _timestamp‚1.784524814868826e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23002019389952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23002019389952 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3054047603831808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3054047245368320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚288 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3013304323483152 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 _step‚1630 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 356245742 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195922845 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3031045584441856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚681334521979904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 109463552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81786465 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Þ ÉT¶C4º=/ _wandbruntime‚17989  _runtime‚17989ö…N'»=N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚681368693369344 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3054196629969408 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160330688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195922358 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22968088904258 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160330690 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 % _timestamp‚1.7845248154835052e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 356253046 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23003129496064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81790184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34542234 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3031193841904640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195932515 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 356263205 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 356263205 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3031193858944000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160330688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚658237130896384 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23002788064768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚1631 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195922358 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34542231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3013451754150624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23003129496064 _runtime‚17989.059299541 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47247953 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3031193858944000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23002788064768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚658237112284160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚19799;ÂÌô'¼=•N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 356263205 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 356263205 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3031193858944000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160330688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚280 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚658237130896384 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23002788064768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 _step‚1631 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195922358 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34542231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3013451754150624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23003129496064 _runtime‚17989.059299541 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47247953 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3031193858944000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23002788064768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚658237112284160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚19799 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚681368693369344 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3054196629969408 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160330688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195922358 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚22968088904258 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160330690 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 % _timestamp‚1.7845248154835052e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 356253046 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23003129496064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81790184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34542234 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3031193841904640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195932515ß ó„X4½=/ _wandbruntime‚17989  _runtime‚17989q8²Y¾=‹ 2loss/SRqqt1/assignment_loss‚0.4100577300149343 1loss/SRqqt2/detection_loss‚0.5994076198973669 epoch‚49 trainer/global_step‚19799 0loss/FRt1/assignment_loss‚1.4341566831325039 0loss/FRt2/assignment_loss‚0.6124247839651912 3loss/SRqqt2/assignment_loss‚0.44786761262606883 0loss/FRt1/detection_loss‚0.46392299686120814 /loss/FRt2/detection_loss‚1.4500089645262313 1loss/SRqqt1/detection_loss‚0.6933301109481942 /loss/FBt2/detection_loss‚0.3406208625251782 %loss/total_loss‚6.806040607963455 4loss/FBt2/assignment_loss‚6.4227129932365165e-06 0loss/FBt1/detection_loss‚0.35372930574995404 _runtime‚17989.059844761 _step‚1632 3loss/FBt1/assignment_loss‚0.0005075150036311869 # _timestamp‚1.78452481548457e+09Æ•]à¿= 0loss/FRt2/assignment_loss‚0.6124247839651912 3loss/SRqqt2/assignment_loss‚0.44786761262606883 0loss/FRt1/detection_loss‚0.46392299686120814 /loss/FRt2/detection_loss‚1.4500089645262313 1loss/SRqqt1/detection_loss‚0.6933301109481942 /loss/FBt2/detection_loss‚0.3406208625251782 %loss/total_loss‚6.806040607963455 4loss/FBt2/assignment_loss‚6.4227129932365165e-06 0loss/FBt1/detection_loss‚0.35372930574995404 _runtime‚17989.059844761 _step‚1632 3loss/FBt1/assignment_loss‚0.0005075150036311869 # _timestamp‚1.78452481548457e+09 2loss/SRqqt1/assignment_loss‚0.4100577300149343 1loss/SRqqt2/detection_loss‚0.5994076198973669 epoch‚49 trainer/global_step‚19799 0loss/FRt1/assignment_loss‚1.4341566831325039à M;\„À=‚(:ù šàöÒ„Ì' proc.memory.availableMB‚ 477.0234375* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17( disk./.usagePercent‚76.1812350160232' disk.nvme0n1.out‚114598.38623046875" disk.nvme1c1n1.in‚ 229.87109375 network.sent‚ 3.2010581e+07 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 313.8515625 disk.md0.out‚ 767.67578125$ memory_percent‚97.08847999572754# proc.memory.rssMB‚ 3025.80078125 cpu‚25.692906531875956 network.recv‚ 4.708723e+06$ disk./.usageGB‚1169.572410583496 disk.dm-1.in‚0" disk.dm-1.out‚16693.8115234375! disk.nvme0c0n1.in‚ 313.8515625( disk.nvme1c1n1.out‚54799.25537109375& disk.nvme1n1.out‚54799.25537109375 disk.md0.in‚0) disk.nvme0c0n1.out‚114598.38623046875 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 543.72265625# disk.dm-0.out‚151095.0830078125át©OœÁ=‚(:‘ šàöÒÔÏÈÌ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚56.0 gpu.0.powerWatts‚281.267' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.75566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 .m„Â=‚(:ù ©àöÒŽçÑË disk.nvme0n1.in‚ 313.8515625' disk.nvme0n1.out‚115261.64990234375" disk.nvme1c1n1.in‚ 229.87109375$ memory_percent‚97.08850383758545( proc.memory.availableMB‚ 477.01953125) disk./.usagePercent‚76.17821009340396 disk.dm-1.in‚0# disk.dm-1.out‚16693.89892578125' disk.nvme1c1n1.out‚55003.9228515625% disk.nvme1n1.out‚55003.9228515625 cpu‚25.69313537447005 network.recv‚ 4.714267e+06% disk./.usageGB‚1169.5259704589844 disk.dm-0.in‚ 543.72265625# disk.dm-0.out‚151962.2587890625 disk.nvme1n1.in‚ 229.87109375# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 network.sent‚ 3.206056e+07 disk.md0.in‚0 disk.rbd0.out‚0) disk.nvme0c0n1.out‚115261.64990234375 proc.cpu.threads‚17 disk.md0.out‚ 767.98046875 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 313.8515625÷8˜žÃ=‚(:“ ©àöÒ¬²Ì gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚58.0 gpu.0.powerWatts‚275.303' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚91.76766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§²†{ƒÄ=‚(:ø ¸àöÒÞ“ËË' disk.nvme0n1.out‚115272.10009765625& disk.nvme1c1n1.out‚55004.388671875 disk.dm-0.in‚ 543.72265625# disk.dm-0.out‚151972.6962890625$ memory_percent‚97.08888530731201! disk.nvme0c0n1.in‚ 313.8515625" disk.nvme1c1n1.in‚ 229.87109375$ disk.nvme1n1.out‚55004.388671875 disk.md0.out‚ 768.17578125( proc.memory.availableMB‚ 476.95703125# proc.memory.rssMB‚ 3025.80078125 cpu‚25.69344600607823 proc.cpu.threads‚17 network.sent‚ 3.2069669e+07 disk.nvme0n1.in‚ 313.8515625 disk.md0.in‚0# disk.dm-1.out‚16693.94580078125* proc.memory.percent‚18.468022346496582 network.recv‚ 4.724218e+06) disk./.usagePercent‚76.16270357575173 disk.rbd0.out‚0 disk.nvme1n1.in‚ 229.87109375 disk.dm-1.in‚0% disk./.usageGB‚1169.2879066467285 disk.rbd0.in‚0) disk.nvme0c0n1.out‚115272.10009765625ˆ䬓Å=‚(:ˆ ¸àöÒ»Ã¡Ì gpu.0.gpu‚99.0 gpu.0.memory‚37, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚121.443' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.481 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O™½Ö4Æ=/ _wandbruntime‚18032  _runtime‚18032¤Sù¤ÊÇ=Ä & lr-AdamW/pg1‚8.998595219235248e-08 & lr-AdamW/pg2‚8.998595219235248e-08 trainer/global_step‚19849 % _timestamp‚1.7845248584509099e+09 _runtime‚18032.026489412 _step‚1633NówÏÈ=É % _timestamp‚1.7845248584509099e+09 _runtime‚18032.026489412 _step‚1633 & lr-AdamW/pg1‚8.998595219235248e-08 & lr-AdamW/pg2‚8.998595219235248e-08 trainer/global_step‚19849á ¨ÊÔ*4É=/ _wandbruntime‚18032  _runtime‚18032ŽÒ§°Ê=†P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160712545 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3038459292509696 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23022804290688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 160712547 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3061516539577344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _runtime‚18032.027457543 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196407497 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 357109899 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23057247067648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 5ú±•m\61386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23057588491776 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23186365179392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34627197 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚659862964130304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107366400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3020675856856752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚298 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3038459309549056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47360608 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3061516539577344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196407497 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚659862945518080 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3020675839817392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23186257812992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47360903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23022463729414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196397354 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357120044 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81987805 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚1634 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚683049329309696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3038459292509696 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23057588491776 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3061516539577344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196407497 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚659862945518080 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3020675839817392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23186257812992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47360903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23022463729414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196397354 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357120044 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81987805 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 _step‚1634 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚683049329309696 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3038459292509696 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23057588491776 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3038459292509696 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23022804290688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚295 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 160712547 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3061516539577344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 _runtime‚18032.027457543 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196407497 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 357109899 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23057247067648 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23057588491776 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23186365179392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2â õûó~4Ì=/ _wandbruntime‚18032  _runtime‚18032œÜÒi'Í=N trainer/global_step‚19849 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47363237 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23058698597888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚1635 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160720340 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3038607584051200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357137507 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196417167 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 $ _timestamp‚1.784524859066331e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160720338 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23058357166592 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3038607567011840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160720338 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81991886 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34628931 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23023572117666 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23187373291008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196407010 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47362955 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23058698597888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _runtime‚18032.642134215 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3038607584051200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3043846842602530 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚659896144400384 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357137507 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 357127348 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196407010 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3043847200203276 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357127348 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272wîе'Î=”N RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3038607584051200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3043846842602530 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚659896144400384 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357137507 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 357127348 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196407010 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3043847200203276 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357127348 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 trainer/global_step‚19849 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47363237 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23058698597888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 _step‚1635 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160720340 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3038607584051200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23995613184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357137507 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196417167 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 $ _timestamp‚1.784524859066331e+09 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160720338 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23058357166592 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3038607567011840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160720338 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81991886 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34628931 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23023572117666 P@DeviceStatsMonitor.on_traÚqó in_batch_end/reserved_bytes.all.current‚ 24566038528 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23187373291008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196407010 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47362955 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23995613184 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23058698597888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 _runtime‚18032.642134215 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496ã éÁû4Ï=/ _wandbruntime‚18032  _runtime‚18032f£Ï—Ð=ˆ 0loss/FRt2/assignment_loss‚0.5827244576336681 2loss/SRqqt2/assignment_loss‚0.4105789350396678 2loss/FBt1/assignment_loss‚0.005944075565668306 1loss/SRqqt1/detection_loss‚0.5775707840708046 /loss/FBt1/detection_loss‚0.2955121871397083 0loss/FBt2/detection_loss‚0.45884051222282535 trainer/global_step‚19849 /loss/FRt1/detection_loss‚0.4510191808984785 _runtime‚18032.642718936 %loss/total_loss‚6.622609723393663 epoch‚49 /loss/FRt2/detection_loss‚1.5066779788571145 $ _timestamp‚1.784524859067345e+09 _step‚1636 3loss/SRqqt1/assignment_loss‚0.38147554459794025 2loss/FBt2/assignment_loss‚0.009838379178040632 1loss/SRqqt2/detection_loss‚0.5510509565940017 0loss/FRt1/assignment_loss‚1.3913767315957448À| <Ñ= 2loss/SRqqt2/assignment_loss‚0.4105789350396678 2loss/FBt1/assignment_loss‚0.005944075565668306 1loss/SRqqt1/detection_loss‚0.5775707840708046 /loss/FBt1/detection_loss‚0.2955121871397083 0loss/FBt2/detection_loss‚0.45884051222282535 trainer/global_step‚19849 /loss/FRt1/detection_loss‚0.4510191808984785 _runtime‚18032.642718936 %loss/total_loss‚6.622609723393663 epoch‚49 /loss/FRt2/detection_loss‚1.5066779788571145 $ _timestamp‚1.784524859067345e+09 _step‚1636 3loss/SRqqt1/assignment_loss‚0.38147554459794025 2loss/FBt2/assignment_loss‚0.009838379178040632 1loss/SRqqt2/detection_loss‚0.5510509565940017 0loss/FRt1/assignment_loss‚1.3913767315957448 0loss/FRt2/assignment_loss‚0.5827244576336681ä ÏB%„Ò=‚(:ù ÇàöÒýÐòË disk.nvme0n1.in‚ 313.8515625) disk./.usagePercent‚76.14724874069395( disk.nvme1c1n1.out‚55005.22509765625# disk.dm-0.out‚151973.6728515625 disk.dm-1.in‚0! disk.dm-1.out‚16694.095703125$ memory_percent‚97.08943367004395# proc.memory.rssMB‚ 3025.80078125 disk.md0.out‚ 768.77734375 disk.dm-0.in‚ 543.72265625 disk.rbd0.out‚0) disk.nvme0c0n1.out‚115273.65380859375* proc.memory.percent‚18.468022346496582 cpu‚25.693810410537882 network.sent‚ 3.2087396e+07 network.recv‚ 4.727642e+06" disk.nvme1c1n1.in‚ 229.87109375! disk.nvme0c0n1.in‚ 313.8515625' disk.nvme0n1.out‚115273.65380859375' proc.memory.availableMB‚ 476.8671875 proc.cpu.threads‚17$ disk./.usageGB‚1169.050636291504 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚55005.22509765625 disk.md0.in‚0 disk.rbd0.in‚0Òê#fžÓ=‚(:“ ÇàöÒÈÕ¯Ì gpu.0.gpu‚60.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚253.018' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚84.33933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÈM-€Ô=‚(:õ ÖàöÒÝæÕË proc.cpu.threads‚17% disk./.usageGB‚1169.1149787902832! disk.dm-1.out‚16694.142578125 disk.rbd0.in‚0 disk.md0.in‚0$ memory_percent‚97.08945751190186) disk./.usagePercent‚76.15143975185039! disk.nvme0c0n1.in‚ 313.8515625 disk.nvme0n1.in‚ 313.8515625( disk.nvme1c1n1.out‚55007.56884765625& disk.nvme1n1.out‚55007.56884765625 disk.md0.out‚ 769.03125$ disk.dm-0.out‚152022.73583984375# proc.memory.rssMB‚ 3025.80078125 network.sent‚ 3.2137798e+07 disk.rbd0.out‚0% disk.nvme0n1.out‚115320.974609375" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 543.72265625( proc.memory.availableMB‚ 476.86328125 cpu‚25.693996015145597 network.recv‚ 4.732971e+06 disk.dm-1.in‚0' disk.nvme0c0n1.out‚115320.974609375* proc.memory.percent‚18.468022346496582oŠ6žÕ=‚(:“ ÖàöÒ”Ì gpu.0.gpu‚60.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚290.302' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚96.76733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«ßª|Ö=‚(:ö åàöÒô©ÀË( disk.nvme0c0n1.out‚115349.4091796875* proc.memory.percent‚18.468022346496582 cpu‚25.69415350578561# disk.dm-1.out‚16694.37060546875 disk.nvme0n1.in‚ 313.8515625& disk.nvme0n1.out‚115349.4091796875" disk.nvme1c1n1.in‚ 229.87109375' proc.memory.availableMB‚ 476.8671875# proc.memory.rssMB‚ 3025.80078125% disk./.usageGB‚1169.1151008605957 disk.dm-0.in‚ 543.72265625 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 313.8515625 disk.md0.in‚0 disk.md0.out‚ 769.484375 proc.cpu.threads‚17 network.sent‚ 3.2142639e+07) disk./.usagePercent‚76.15144770301877$ disk.dm-0.out‚152050.73193359375 disk.dm-1.in‚0' disk.nvme1c1n1.out‚55008.3369140625 disk.nvme1n1.in‚ 229.87109375% disk.nvme1n1.out‚55008.3369140625$ memory_percent‚97.08943367004395 network.recv‚ 4.736138e+06 disk.rbd0.out‚0/é2×=‚(:’ åàöÒÛ«Ì gpu.0.gpu‚27.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚280.144' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚93.38133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?}P:4Ø=/ _wandbruntime‚18075  _runtime‚18075š;îÊÙ=Ä & lr-AdamW/pg1‚4.026014040244181e-08 & lr-AdamW/pg2‚4.026014040244181e-08 trainer/global_step‚19899 % _timestamp‚1.7845249018930016e+09 _runtime‚18075.468643829 _step‚1637rœ«ÊÏÚ=É & lr-AdamW/pg1‚4.026014040244181e-08 & lr-AdamW/pg2‚4.026014040244181e-08 trainer/global_step‚19899 % _timestamp‚1.7845249018930016e+09 _runtime‚18075.468643829 _step‚1637å Ëob¿4Û=/ _wandbruntime‚18075  _runtime‚18075Fí¸Ë (Ü=†P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3045873017616896 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3045873034656256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚684764045475328 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82189899 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196892149 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3028047390230352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3068985833786368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23112816169472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23113157593600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23242086453248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚661521977634304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357994346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34713900 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23113157593600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚661521959022080 _step‚1638 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23077946942822 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 357984201 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161102197 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23112816169472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3068985833786368 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196882006 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3068986192249856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚684764177745408 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3045873017616896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196882006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161102197 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47475999 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161102195 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864”‚„(Ý=‹P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3045873017616896 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3045873034656256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚684764045475328 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82189899 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196892149 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3028047390230352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1180 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3068985833786368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23112816169472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23113157593600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23242086453248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚299 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚661521977634304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 113657856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357994346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34713900 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23113157593600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚661521959022080 _step‚1638 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23077946942822 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 357984201 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161102197 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23112816169472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 626354176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3068985833786368 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196882006 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3068986192249856 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚684764177745408 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3045873017616896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196882006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161102197 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47475999 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161102195 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 132270080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272æ ÏR 4Þ=/ _wandbruntime‚18076  _runtime‚18076>¦†ùŠ ß=N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82193848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358470656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 128068608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚23079395892460 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161109990 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚1639 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196901819 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196901819 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34715631 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23243311676416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196891662 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3028194803858464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23079055331074 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358011809 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196891662 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358001650 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚661555139292160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23114267699712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82193582 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23243202220032 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3069135576858112 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3046021292119040 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23113926268416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _step‚1639 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196901819 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196901819 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34715631 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23243311676416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196891662 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23343568896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 626354176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3028194803858464 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23079055331074 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358011809 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196891662 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358001650 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚661555139292160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23114267699712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82193582 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23243202220032 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3069135576858112 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3046021292119040 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23113926268416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358858503 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23133770717504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3053286759763456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3053286759763456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3058552336720822 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3058552694321456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23168726695424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚663180991138304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚663180972526080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23133430156230 trainer/global_step‚19949 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82390991 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197366658 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34800597 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚18119.626697517 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3076455486458880 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚686478979321344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3076455486458880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82390707 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161491847 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3053286742724096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23297885010944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3076455127995392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _step‚1642 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3035418906564592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47590391 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47590110 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23168385271296 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197376801 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23168385271296 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23297988183040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 $ _timestamp‚1.784524946050831e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358868648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161491845 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚686478857537024 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197366658 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34800597 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 _runtime‚18119.626697517 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3076455486458880 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚686478979321344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚272 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3076455486458880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82390707 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161491847 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3053286742724096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23297885010944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3076455127995392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24566038528 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1452 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 _step‚1642 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3035418906564592 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341424128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47590391 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47590110 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚23168385271296 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197376801 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23168385271296 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚23297988183040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23995613184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 $ _timestamp‚1.784524946050831e+09 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358868648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161491845 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23995613184 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 570425344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚686478857537024 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103172096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358858503 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚23133770717504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3053286759763456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121784320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3053286759763456 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3058552336720822 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1452 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 717119488 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3058552694321456 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚23168726695424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚663180991138304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚663180972526080 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚23133430156230 trainer/global_step‚19949 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82390991 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3053435017226240 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23169836801536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚663214152796160 % _timestamp‚1.7845249466750865e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚19949 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3058701233377292 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358886111 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197376314 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47592625 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23299103180800 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23169836801536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3053435034265600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23134538544482 _runtime‚18120.250887328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82394699 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197376314 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3035566337232064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚686513150714880 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47592368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23169495370240 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3053435034265600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23298997918720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34802334 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0\²†X'ò=•N F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47592368 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105262080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23169495370240 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3053435034265600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23298997918720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34802334 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358875952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82394959 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3053435017226240 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23169836801536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚663214152796160 % _timestamp‚1.7845249466750865e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚19949 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123874304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3058701233377292 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358886111 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197376314 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47592625 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23299103180800 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560516799 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 561386496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23169836801536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24566038528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24566038528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3053435034265600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23134538544482 _runtime‚18120.250887328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82394699 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197376314 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3035566337232064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚686513150714880 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180ë ,Î"4ó=/ _wandbruntime‚18120  _runtime‚18120OU‘cô=ˆ 2loss/FBt1/assignment_loss‚0.004905679791744843 1loss/SRqqt1/detection_loss‚0.7039298418795141 0loss/FRt1/assignment_loss‚1.4979320757736148 0loss/FRt2/assignment_loss‚0.6575594376295041 2loss/SRqqt2/assignment_loss‚0.4237280156972132 $loss/total_loss‚7.06065221687323 % _timestamp‚1.7845249466762176e+09 3loss/SRqqt1/assignment_loss‚0.41920205694972057 3loss/FBt2/assignment_loss‚0.0014308813335530443 /loss/FRt1/detection_loss‚0.4656354470745255 /loss/FBt1/detection_loss‚0.4238634780801828 0loss/FBt2/detection_loss‚0.37676497453616725 epoch‚49 _step‚1644 /loss/FRt2/detection_loss‚1.4445619735278468 1loss/SRqqt2/detection_loss‚0.6411383545996436 trainer/global_step‚19949 _runtime‚18120.25170299Ÿ¿âF3õ= 0loss/FBt2/detection_loss‚0.37676497453616725 epoch‚49 _step‚1644 /loss/FRt2/detection_loss‚1.4445619735278468 1loss/SRqqt2/detection_loss‚0.6411383545996436 trainer/global_step‚19949 _runtime‚18120.25170299 2loss/FBt1/assignment_loss‚0.004905679791744843 1loss/yH½DàSRqqt1/detection_loss‚0.7039298418795141 0loss/FRt1/assignment_loss‚1.4979320757736148 0loss/FRt2/assignment_loss‚0.6575594376295041 2loss/SRqqt2/assignment_loss‚0.4237280156972132 $loss/total_loss‚7.06065221687323 % _timestamp‚1.7845249466762176e+09 3loss/SRqqt1/assignment_loss‚0.41920205694972057 3loss/FBt2/assignment_loss‚0.0014308813335530443 /loss/FRt1/detection_loss‚0.4656354470745255 /loss/FBt1/detection_loss‚0.4238634780801828ì Kq ¢}ö=‚(:ò ¡áöÒç÷½Ë network.recv‚ 4.753205e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚ 313.8515625" disk.nvme1c1n1.in‚ 229.87109375& disk.nvme1n1.out‚55119.76025390625" proc.memory.availableMB‚476.75* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17) disk./.usagePercent‚76.16281017110278% disk.nvme0n1.out‚115949.763671875 disk.md0.in‚0 cpu‚25.69362196729791 network.sent‚ 3.2264805e+07 disk.dm-1.in‚0" disk.dm-1.out‚16695.0283203125! disk.nvme0c0n1.in‚ 313.8515625( disk.nvme1c1n1.out‚55119.76025390625 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 771.08984375$ memory_percent‚97.09014892578125% disk./.usageGB‚1169.2895431518555 disk.rbd0.out‚0' disk.nvme0c0n1.out‚115949.763671875 disk.dm-0.in‚ 543.72265625$ disk.dm-0.out‚152758.39892578125# proc.memory.rssMB‚ 3025.80078125zî’“÷=‚(:ˆ ¡áöÒüˆ°Ì gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚57.0 gpu.0.powerWatts‚278.151' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚92.717 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ié®xø=‚(:í °áöÒæöÀË" disk.dm-1.out‚16695.0712890625 disk.nvme0n1.in‚ 313.8515625" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375& disk.nvme1n1.out‚55122.64501953125" proc.memory.availableMB‚476.75( disk./.usagePercent‚76.1472728426731 disk.dm-0.in‚ 543.72265625$ memory_percent‚97.09014892578125# proc.memory.rssMB‚ 3025.80078125 proc.cpu.threads‚17$ disk.dm-0.out‚152774.92626953125! disk.nvme0c0n1.in‚ 313.8515625# disk.nvme0n1.out‚115964.0703125* proc.memory.percent‚18.468022346496582 cpu‚25.693393202450093% disk.nvme0c0n1.out‚115964.0703125 network.sent‚ 3.231856e+07 disk.md0.in‚0 disk.md0.out‚ 771.37109375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme1c1n1.out‚55122.64501953125 network.recv‚ 4.758743e+06% disk./.usageGB‚1169.0510063171387šöÒ‘ù=‚(:† °áöÒÐë‰Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.312' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?êhÕ4ú=/ _wandbruntime‚18163  _runtime‚181632ùPÉû=à & lr-AdamW/pg2‚3.946771832389151e-12 trainer/global_step‚19999 $ _timestamp‚1.784524990392789e+09 _runtime‚18163.968376643 _step‚1645 & lr-AdamW/pg1‚3.946771832389151e-12u€Îü=È _runtime‚18163.968376643 _step‚1645 & lr-AdamW/pg1‚3.946771832389151e-12 & lr-AdamW/pg2‚3.946771832389151e-12 trainer/global_step‚19999 $ _timestamp‚1.784524990392789e+09í :CÎ4ý=/ _wandbruntime‚18163  _runtime‚18163lJ×— (þ=†P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358463488 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161881495 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23995613184 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34887297 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 561386496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197861453 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚664839986030080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47704573 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚511 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23175647232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560516799 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1180 _step‚1646 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23343568896 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1452 trainer/global_step‚19999 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚23224295797248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82592163 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 197861453 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3060700484870656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1534 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚272 _runtime‚18163.969424386 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚23223954373120 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚664840004642304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3083924780667904 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3060700467831296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3065979353307830 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23873614336 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161881497 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82591870 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197851310 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3065979710908464 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688193820447744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3060700484870656 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3083924780667904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3042790439938192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23353710536192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3083924422204416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161881497 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161881495 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3065979710908464 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23343568896 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚688193820447744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3060700484870656 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358463488 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3083924780667904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3042790439938192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24566038528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚23353710536192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23704151647 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3083924422204416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 123881472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 174180864 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 570425344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚85 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161881497 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1180 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 561386496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161881495 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341424128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 570425344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1534 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 105269248 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24566038528 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23873614336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340561274 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161881497 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82591870 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚290 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357600634 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197851310 J/ _wandbruntime‚18164  _runtime‚18164ÂT…®\!>ŽN \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3060848759372800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197871123 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚18164.958106843 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3084074165276160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23354934159872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161889290 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197860966 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47706826 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23225405903360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 359750254 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82596141 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197860966 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23225064472064 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47707107 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23225064472064 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3066128249964300 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23190021757890 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 359750254 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23225405903360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161889288 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3066127892363554 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688228119072256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82595857 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23354820509184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 # _timestamp‚1.78452499138232e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3084073806805504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34889034 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 359760413 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 359760413 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚664873184912384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3060848742333440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3042937870605664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85øU‰¼'‚>“N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34889031 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚688227986809344 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197871123 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 132262912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1180 _runtime‚18164.958106843 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3084074165276160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚23354934159872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23175647232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1180 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358470656 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161889290 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197860966 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47706826 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚281 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 174180864 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1452 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1180 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚272 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23873614336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚23225405903360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 359750254 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23704151647 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82596141 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24566038528 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197860966 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚23225064472064 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23873614336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47707107 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚272 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 570425344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23343568896 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚23225064472064 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3066128249964300 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340561386 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚23190021757890 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 359750254 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚23225405903360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161889288 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3066127892363554 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚511 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688228119072256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82595857 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 113650688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚23354820509184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 570425344 # _timestamp‚1.78452499138232e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3084073806805504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34889034 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 359760413 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 717119488 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 359760413 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341431296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 561386496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1534 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚664873184912384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 570425344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3060848742333440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3042937870605664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚85 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3060848759372800 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341431296 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23995613184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚284 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357600746 F/ _wandbruntime‚18164  _runtime‚18164Kò{„>‰ % _timestamp‚1.7845249913835177e+09 _step‚1648 0loss/FRt1/assignment_loss‚1.6149846069041016 0loss/FRt2/assignment_loss‚0.5491572700294793 1loss/SRqqt1/assignment_loss‚0.413071992134857 1loss/FBt1/assignment_loss‚0.00471157144890268 1loss/SRqqt1/detection_loss‚0.5980163921046001 %loss/total_loss‚6.699210952718148 epoch‚49 _runtime‚18164.958865515 /loss/FRt2/detection_loss‚1.4235275165148207 0loss/FBt2/detection_loss‚0.27629156195190413 2loss/SRqqt2/assignment_loss‚0.4751693129116287 4loss/FBt2/assignment_loss‚1.4610829093904814e-05 /loss/FRt1/detection_loss‚0.4387141939209492 1loss/SRqqt2/detection_loss‚0.5749759480878444 0loss/FBt1/detection_loss‚0.33057597587996596 trainer/global_step‚19999ºë…>Ž 0loss/FRt1/assignment_loss‚1.6149846069041016 0loss/FRt2/assignment_loss‚0.5491572700294793 1loss/SRqqt1/assignment_loss‚0.413071992134857 1loss/FBt1/assignment_loss‚0.00471157144890268 1loss/SRqqt1/detection_loss‚0.5980163921046001 %loss/total_loss‚6.699210952718148 epoch‚49 _runtime‚18164.958865515 /loss/FRt2/detection_loss‚1.4235275165148207 0loss/FBt2/detection_loss‚0.27629156195190413 2loss/SRqqt2/assignment_loss‚0.4751693129116287 4loss/FBt2/assignment_loss‚1.4610829093904814e-05 /loss/FRt1/detection_loss‚0.4387141939209492 1loss/SRqqt2/detection_loss‚0.5749759480878444 0loss/FBt1/detection_loss‚0.33057597587996596 trainer/global_step‚19999 % _timestamp‚1.7845249913835177e+09 _step‚1648ð –úäã…†>‚(:ú ¿áöÒ‘š”Ë# proc.memory.rssMB‚ 3025.80078125* proc.memory.percent‚18.468022346496582 proc.cpu.threads‚17 network.recv‚ 4.768431e+06 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 313.8515625) disk.nvme0c0n1.out‚116044.51611328125 disk.nvme0n1.in‚ 313.8515625' disk.nvme0n1.out‚116044.51611328125" disk.nvme1c1n1.in‚ 229.87109375 disk.md0.in‚0) disk./.usagePercent‚76.16303777329756% disk.nvme1n1.out‚55212.7861328125 disk.md0.out‚ 771.73046875$ memory_percent‚97.09165096282959( proc.memory.availableMB‚ 476.50390625 network.sent‚ 3.2325513e+07% disk./.usageGB‚1169.2930374145508' disk.nvme1c1n1.out‚55212.7861328125 disk.nvme1n1.in‚ 229.87109375 disk.dm-0.in‚ 543.72265625$ disk.dm-0.out‚152944.56298828125 cpu‚25.69317967849044 disk.dm-1.in‚0" disk.dm-1.out‚16695.2392578125 disk.rbd0.in‚0—Ë]œ‡>‚(:‘ ¿áöÒæ­‰Ì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚55.0 gpu.0.powerWatts‚120.359' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚40.11966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À†YÅ‚ˆ>‚(:÷ ÎáöÒ÷ä×Ë disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 543.72265625# proc.memory.rssMB‚ 3024.48046875) proc.memory.percent‚18.45996379852295 network.sent‚ 3.2389063e+07( disk.nvme1c1n1.out‚55214.51513671875& disk.nvme1n1.out‚55214.51513671875" disk.dm-1.out‚16695.3564453125) disk.nvme0c0n1.out‚116056.97900390625 cpu‚25.695175916434952 proc.cpu.threads‚17" disk.nvme1c1n1.in‚ 229.87109375 disk.nvme1n1.in‚ 229.87109375 disk.md0.out‚ 772.03515625 disk.dm-0.out‚152957.9609375 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 313.8515625$ memory_percent‚97.08378314971924( proc.memory.availableMB‚ 477.79296875 network.recv‚ 4.772871e+06) disk./.usagePercent‚76.16303429466139$ disk./.usageGB‚1169.292984008789 disk.nvme0n1.in‚ 313.8515625' disk.nvme0n1.out‚116056.97900390625ì/4FŸ‰>‚(:” ÎáöÒÌ‘èË gpu.0.gpu‚43.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚29.239273071289062* gpu.0.memoryAllocatedBytes‚ 25116344320 gpu.0.temp‚54.0 gpu.0.powerWatts‚121.183' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.394333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Óò}´4Š>/ _wandbruntime‚18181  _runtime‚18181Yj`Ð#=‹>z 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0759354680776596 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6825920939445496 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_2_of_2‚0.15996456146240234 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05745203047990799 epoch‚49 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5787264704704285 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21380268037319183 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0672011747956276 particle/accuracy_5_of_5‚1 _step‚1649 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6237191557884216 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6897379755973816 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6413515210151672 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.347939670085907 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6233543157577515 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07976909726858139 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6465515494346619 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5053138136863708 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05017391964793205 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7757686376571655 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9695299863815308 /particle/accuracy_3_of_3‚0.9994120597839355 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5971326231956482 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28969424962997437 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7106602787971497 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5772291421890259 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30013135075569153 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09116977453231812 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2181686908006668 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 )loss/val_total_loss‚7.424136646332054 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20979444682598114 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5812360644340515 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_3_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2937626540660858 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7051336169242859 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33854278922080994 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31403595209121704 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7525046467781067 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5669607520103455 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6954892873764038 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3000299036502838 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9942034482955933 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6253431439399719 (particle/accuracy‚0.8530119061470032 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5995126962661743 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5971373915672302 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22943007946014404 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5772291421890259 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3436606824398041 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07644981890916824 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7251509428024292 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6355450749397278 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6475217342376709 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7757686376571655 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2386254072189331 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.48121288418769836 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.667098343372345 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.715973973274231 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6574587821960449 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_1‚1 jet/accuracy_5_of_5‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5976540446281433 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2640339732170105 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2686667740345001 /particle/accuracy_4_of_4‚0.9579095244407654 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9178250432014465 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5653565526008606 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8410053253173828 particle/accuracy_1_of_3‚1 +particle/specificity‚0.9107475280761719 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9090608954429626 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997935950756073 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04707628861069679 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27399659156799316 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11160778999328613 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.37182846665382385 jet/accuracy_1_of_5‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2929379642009735 +jet/accuracy_3_of_3‚0.07523223012685776 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8866800665855408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5773653388023376 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6485018730163574 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28924253582954407 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21736519038677216 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9488723278045654 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7193347215652466 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6219285130500793 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7296767830848694 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5708333253860474 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6237139105796814 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.28382691740989685 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7719518542289734 particle/accuracy_1_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861111044883728 *jet/accuracy_2_of_4‚0.7296527028083801 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47709497809410095 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30140766501426697 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28924253582954407 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.08844424784183502 +particle/sensitivity‚0.6419110894203186 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.294146865606308 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8641345500946045 particle/accuracy_2_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9931437373161316 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0672011747956276 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2279699593782425 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07894910126924515 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30268871784210205 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4670901894569397 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6338709592819214 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8662438988685608 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5750190019607544 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9981684684753418 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26283395290374756 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4799939692020416 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7069956660270691 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4462648630142212 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2746613919734955 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.252913236618042 +jet/accuracy_1_of_1‚0.33098462224006653 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3452839255332947 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2806803286075592 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8389833569526672 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03918593004345894 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9975854754447937 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.6522465348243713 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2266470342874527 )jet/accuracy_2_of_3‚0.555601954460144 trainer/global_step‚19999 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6637673377990723 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.668846607208252 % _timestamp‚1.7845250080189822e+09 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9979742765426636 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6158626675605774 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_2_of_5‚1 _runtime‚18181.595399381'ô{ëŒ>¢z 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6237139105796814 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.28382691740989685 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7719518542289734 particle/accuracy_1_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861111044883728 *jet/accuracy_2_of_4‚0.7296527028083801 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47709497809410095 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30140766501426697 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28924253582954407 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.08844424784183502 +particle/sensitivity‚0.6419110894203186 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.294146865606308 7 Purity/*FRt0SRqq’vj=8t2FBt/FRt_purity‚0.8641345500946045 particle/accuracy_2_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9931437373161316 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0672011747956276 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.782974362373352 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2279699593782425 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07894910126924515 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9974821209907532 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.30268871784210205 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4670901894569397 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6338709592819214 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.7213542461395264 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8662438988685608 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5750190019607544 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9981684684753418 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.26283395290374756 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4799939692020416 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7069956660270691 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4462648630142212 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2746613919734955 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.252913236618042 +jet/accuracy_1_of_1‚0.33098462224006653 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3452839255332947 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2806803286075592 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8389833569526672 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03918593004345894 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981483578681946 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9975854754447937 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.6522465348243713 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2266470342874527 )jet/accuracy_2_of_3‚0.555601954460144 trainer/global_step‚19999 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6637673377990723 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.668846607208252 % _timestamp‚1.7845250080189822e+09 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9979742765426636 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6158626675605774 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_2_of_5‚1 _runtime‚18181.595399381 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0759354680776596 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6825920939445496 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_2_of_2‚0.15996456146240234 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.05745203047990799 epoch‚49 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5787264704704285 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21380268037319183 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0672011747956276 particle/accuracy_5_of_5‚1 _step‚1649 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6237191557884216 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6897379755973816 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.857350766658783 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6413515210151672 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.6875 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.347939670085907 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6233543157577515 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9937466979026794 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07976909726858139 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6465515494346619 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5053138136863708 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05017391964793205 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7757686376571655 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9695299863815308 /particle/accuracy_3_of_3‚0.9994120597839355 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5971326231956482 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28969424962997437 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7106602787971497 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5772291421890259 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30013135075569153 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09116977453231812 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9874933958053589 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2181686908006668 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.857350766658783 )loss/val_total_loss‚7.424136646332054 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20979444682598114 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5812360644340515 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4739583432674408 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_3_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9980471730232239 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2937626540660858 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7051336169242859 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.33854278922080994 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43287038803100586 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.31403595209121704 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7525046467781067 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5669607520103455 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6954892873764038 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3000299036502838 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9942034482955933 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6253431439399719 (particle/accuracy‚0.8530119061470032 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8928571343421936 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5995126962661743 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5971373915672302 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22943007946014404 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5772291421890259 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3436606824398041 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07644981890916824 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7251509428024292 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.21416668593883514 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9979092478752136 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6355450749397278 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6475217342376709 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7757686376571655 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2386254072189331 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.48121288418769836 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.667098343372345 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.715973973274231 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6574587821960449 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_1‚1 jet/accuracy_5_of_5‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9294871091842651 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7857142686843872 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5976540446281433 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2640339732170105 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2686667740345001 /particle/accuracy_4_of_4‚0.9579095244407654 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9939485192298889 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9178250432014465 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.84375 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5653565526008606 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8410053253173828 particle/accuracy_1_of_3‚1 +particle/specificity‚0.9107475280761719 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9090608954429626 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.997935950756073 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.04707628861069679 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27399659156799316 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.11160778999328613 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.37182846665382385 jet/accuracy_1_of_5‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2929379642009735 +jet/accuracy_3_of_3‚0.07523223012685776 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8866800665855408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5773653388023376 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6485018730163574 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.28924253582954407 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21736519038677216 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9488723278045654 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7193347215652466 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6219285130500793 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7296767830848694 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5708333253860474ñ ÷’1Q> nucmi0u1j? ÐáöÒà‰æ0`Trainer.fit` stopped: `max_epochs=50` reached. ‘wCA¡Ž> nucmi0u1jŽ ÐáöÒèâÒ|Epoch 49/49 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 400/400 0:05:48 • 0:00:00 1.11it/s v_num: i0u1øŸn!%> nucmi0u1j ÐáöÒ ”ô 8âŽô4>/ _wandbruntime‚18184  _runtime‚18184‘¼Ÿ‘>‚(’ˆŽ